search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
23.16k stars 4.67k forks source link

Docker push EOF #20405

Open letran3691 opened 2 months ago

letran3691 commented 2 months ago

I deployed Harbor v2.10.2-1a741cb7 on NAS Synology OS 7.1.1. After deploying Harbor, I can access the web GUI normally. However, when I try to push Docker images, it returns EOF. image I've read the Harbor logs and searched Google; however, the issue remains unresolved image

I hope someone can help me. Thanks.

MinerYang commented 2 months ago

Hi @letran3691 ,

Could you provide more contexts, like proxy logs, harbor-registry logs, docker daemon logs?

letran3691 commented 2 months ago

As mentioned above, I deployed Harbor on a NAS Synology device with the following configurations:

Hi @letran3691 ,

Could you provide more contexts, like proxy logs, harbor-registry logs, docker daemon logs?

MinerYang commented 2 months ago

Hi @letran3691 ,

I haven't' see any request at the harbor proxy about the pushing image harbor-registry.mbageas.life/test/test:latest

letran3691 commented 2 months ago

Hi @letran3691 ,

I haven't' see any request at the harbor proxy about the pushing image harbor-registry.mbageas.life/test/test:latest

  • What's your storage backend? filesystem or s3?
  • Is there any proxy/firewall in front of Harbor or docker service?
  • Could you check the docker daemon logs while you occurring this issue?

"May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.505245247+07:00" level=error msg="Upload failed, retrying: EOF" May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.505417124+07:00" level=error msg="Upload failed, retrying: EOF" May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.505763625+07:00" level=debug msg="Assembling tar data for 3f75b4eed68d743d9d4e5fbba18337bc81f01eeb6914045990b046f27b7c8ac8" May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.509833916+07:00" level=error msg="Upload failed, retrying: EOF" May 10 10:18:53 master01 systemd[1]: var-lib-docker-overlay2-3f75b4eed68d743d9d4e5fbba18337bc81f01eeb6914045990b046f27b7c8ac8-merged.mount: Deactivated successfully. May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.838300586+07:00" level=debug msg="Assembling tar data for 53c15b8073b7e13ff003b0d662a094eddbe59f996aed6187bc896f1bea089efd" May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.838846713+07:00" level=error msg="Upload failed, retrying: EOF" May 10 10:18:54 master01 dockerd[2526225]: time="2024-05-10T10:18:54.049991205+07:00" level=debug msg="Assembling tar data for kvuf2havlofb8kvvpmyek8ufc" May 10 10:18:54 master01 dockerd[2526225]: time="2024-05-10T10:18:54.053474797+07:00" level=error msg="Upload failed, retrying: EOF" May 10 10:18:54 master01 systemd[1]: run-containerd-runc-k8s.io-a21a810c9e7c695423b5a26095f4775bc739f6f1bae53bcaef4e3eff9fa89bce-runc.TEfViB.mount: Deactivated successfully. May 10 10:18:58 master01 systemd[1]: run-containerd-runc-k8s.io-64e9c11c66e28b4a6cfff2cb48e075b99f6aac368f55e3a2eab88fc08b8b9e58-runc.fmBacv.mount: Deactivated successfully. May 10 10:19:00 master01 systemd[1]: run-containerd-runc-k8s.io-a21a810c9e7c695423b5a26095f4775bc739f6f1bae53bcaef4e3eff9fa89bce-runc.QfXN5i.mount: Deactivated successfully. May 10 10:19:04 master01 systemd[1]: run-containerd-runc-k8s.io-a21a810c9e7c695423b5a26095f4775bc739f6f1bae53bcaef4e3eff9fa89bce-runc.0xhSK7.mount: Deactivated successfully. May 10 10:19:08 master01 systemd[1]: run-containerd-runc-k8s.io-64e9c11c66e28b4a6cfff2cb48e075b99f6aac368f55e3a2eab88fc08b8b9e58-runc.sPzWDz.mount: Deactivated successfully. May 10 10:19:10 master01 systemd[1]: run-containerd-runc-k8s.io-a21a810c9e7c695423b5a26095f4775bc739f6f1bae53bcaef4e3eff9fa89bce-runc.STbyul.mount: Deactivated successfully. May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.506184169+07:00" level=debug msg="Pushing layer: sha256:b80974699bdb622b5cbb7ae31868637c5dc1d2ca6a5f61dbe654e4abb2e14632" May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.506184285+07:00" level=debug msg="Pushing layer: sha256:2e0d92876e9f41196bbbb78eabd81a24ab66d43db4d681d910fba3b4a5bd7aa7" May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.510738640+07:00" level=debug msg="Pushing layer: sha256:d75fd41110a448d9d52d83f6e415a4167f3ab4691a7e4948101cf64159cd1e96" May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.839253494+07:00" level=debug msg="Pushing layer: sha256:3f69a32a2b313adbf3d5076a3be902a667b04783a3cdfb7aef74b1b62b0a8df4" May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.935367150+07:00" level=debug msg="Assembling tar data for f00a150a4aef8d77e201fd0100bdc09fe513bd86c4254aa37e018cb03e94c6c1" May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.941133148+07:00" level=error msg="Upload failed: EOF" May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.942887625+07:00" level=debug msg="Assembling tar data for c79454a44ddc7a1f059fa157d4806c718a1862300a7a6fc2d9c460bed9c2b3b7" May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.944553727+07:00" level=error msg="Upload failed: EOF" May 10 10:19:13 master01 systemd[1]: var-lib-docker-overlay2-f00a150a4aef8d77e201fd0100bdc09fe513bd86c4254aa37e018cb03e94c6c1-merged.mount: Deactivated successfully. May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.948713310+07:00" level=debug msg="Checking for presence of layer sha256:bf6cd01dee5edc1e179aeab7de1d24dd401c2cc798cdc61323030f8da0b778ee (sha256:4d84de5fb9b2d32bc417c2e043fcc88e3303970c61cd4916956e66431a9ce7d3) in harbor-registry.mbageas.life/test/login" May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.949205790+07:00" level=debug msg="Checking for presence of layer sha256:a8a738834a4d15fbdc977804f7ed941d23b26b3ccd0364b390acb684a42fed24 (sha256:2818b7b6a9db4d0901d74850cf1096f76fa22d8a7fad4d711710fbf6ab1b48d1) in harbor-registry.mbageas.life/test/login" May 10 10:19:13 master01 systemd[1]: var-lib-docker-overlay2-c79454a44ddc7a1f059fa157d4806c718a1862300a7a6fc2d9c460bed9c2b3b7-merged.mount: Deactivated successfully. May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.968572425+07:00" level=debug msg="Pushing layer: sha256:bf6cd01dee5edc1e179aeab7de1d24dd401c2cc798cdc61323030f8da0b778ee" May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.970249678+07:00" level=debug msg="Pushing layer: sha256:a8a738834a4d15fbdc977804f7ed941d23b26b3ccd0364b390acb684a42fed24" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.053908318+07:00" level=debug msg="Pushing layer: sha256:ecbd97428477513a42d7c54fa91b03e377cb232041b4bbbb4484d56857b3a8cc" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.147648677+07:00" level=debug msg="Assembling tar data for 3f75b4eed68d743d9d4e5fbba18337bc81f01eeb6914045990b046f27b7c8ac8" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.148546806+07:00" level=error msg="Upload failed: EOF" May 10 10:19:14 master01 systemd[1]: var-lib-docker-overlay2-3f75b4eed68d743d9d4e5fbba18337bc81f01eeb6914045990b046f27b7c8ac8-merged.mount: Deactivated successfully. May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.192549944+07:00" level=debug msg="Checking for presence of layer sha256:52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061 (sha256:b0a0cf830b12453b7e15359a804215a7bcccd3788e2bcecff2a03af64bbd4df7) in harbor-registry.mbageas.life/test/login" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.215414543+07:00" level=debug msg="Pushing layer: sha256:52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.643971107+07:00" level=debug msg="Assembling tar data for 53c15b8073b7e13ff003b0d662a094eddbe59f996aed6187bc896f1bea089efd" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.645925541+07:00" level=error msg="Upload failed: EOF" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.811944419+07:00" level=error msg="Upload failed, retrying: EOF" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.818049736+07:00" level=debug msg="Assembling tar data for 2e3ac69c2e08ae61a2441df39d27051280c3cf7b62193097124fc6104d06630c" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.818451492+07:00" level=error msg="Upload failed, retrying: EOF" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.829230060+07:00" level=debug msg="Assembling tar data for 244b5054ff59424ea3fe13c89f35253cb4e83589be3e5e65a5c83875ddca6ceb" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.927658513+07:00" level=error msg="Upload failed: EOF" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.928044306+07:00" level=debug msg="Assembling tar data for kvuf2havlofb8kvvpmyek8ufc" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.936099625+07:00" level=info msg="Attempting next endpoint for push after error: EOF" May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.941759145+07:00" level=debug msg="Assembling tar data for dd1425f918c1c4009babe0bba986e304d40da0d71e8880e46e18d554897fcdb7"

wy65701436 commented 2 months ago

please share the log of registry.

letran3691 commented 2 months ago

please share the log of registry.

it's here registry.log

letran3691 commented 2 months ago

So, does anyone have any ideas?

MinerYang commented 2 months ago

Hi @letran3691 ,

Could you check the blob existence on your filesystem(registry storage)?

letran3691 commented 2 months ago

Hi @letran3691 ,

Could you check the blob existence on your filesystem(registry storage)?

I've checked, and nothing exists. image

MinerYang commented 2 months ago

Hi @letran3691 , Did the push issue only happened in the repository/artifact or generally for all projects?

127.0.0.1:6379[1]> keys "repository:*"

And one more thing, the logs indicates Harbor instance is on `v2.7.3`, Not `v2.10.2-1a741cb7`, Please make sure the log is consistent harbor instance you are using

May 9 12:45:37 192.168.208.1 core[9288]: 2024-05-09T05:45:37Z [INFO] [/core/main.go:259]: Version: v2.7.3, Git commit: 252a0b7c

letran3691 commented 2 months ago

Hi @letran3691 , Did the push issue only happened in the repository/artifact or generally for all projects?

  • If it is only happened for some of the specific image that you customize, then my best guess is it occurs at docker client side when process tar file, please check you the image fileyou build is not corrupted. Since the docker daemon log and the registry logs you provided is not relevant to a same artifact, that I didn't see any harbor-registry.mbageas.life/test/test:latest upload request arrived at the harbor egistry log.
  • If it occurs for all the images including official docker image, then please try to disable all the FW and VPN etc.. (BTW, harbor only official shipped linux amd64 build, We are not testified on NAS Synology OS yet)
  • Please double check with the image test/login:<sha256:123XXX> both in filesystem and redis cache cat /docker/harbor/registry/docker/registry/v2/sha245/12//data
  • docker exec -it /bin/bash
$ redis-cli -n 1
127.0.0.1:6379[1]> keys "blobs:*"

127.0.0.1:6379[1]> keys "repository:*"

And one more thing, the logs indicates Harbor instance is on v2.7.3, Not v2.10.2-1a741cb7, Please make sure the log is consistent harbor instance you are using

May  9 12:45:37 192.168.208.1 core[9288]: 2024-05-09T05:45:37Z [INFO] [/core/main.go:259]: Version: v2.7.3, Git commit: 252a0b7c

it occurs for all the images including official docker image.

this path not exist cat /docker/harbor/registry/docker/registry/v2/sha245/12/data Only the path exists. image

The logs are different because I tried deploying multiple versions, but all have the same issue.

malivix commented 1 month ago

we are facing same issue and I am getting these errors

#87 [python] exporting to image
#87 492.6 error: failed commit on ref "layer-sha256:e003ff9c92020b3ada8328059032e5c69e6e4d5175578b6dd99c844beca49f0d": failed to do request: Put "https://registry.glss.ir/v2/jenkins/builder-python/blobs/uploads/c7f1c997-b4e1-4cfa-8d77-178231e3e067?_state=xsr8xxrIIzshTuDHW-BUk-72zbqwRXToaupsLR4B5CJ7Ik5hbWUiOiJqZW5raW5zL2J1aWxkZXItcHl0aG9uIiwiVVVJRCI6ImM3ZjFjOTk3LWI0ZTEtNGNmYS04ZDc3LTE3ODIzMWUzZTA2NyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyNC0wNi0wMVQyMjo0MToxNy42NjExMzA0MjFaIn0%3D&digest=sha256%3Ae003ff9c92020b3ada8328059032e5c69e6e4d5175578b6dd99c844beca49f0d": EOF
#87 492.6 retrying in 1s

harbor-core-69d86b8495.log harbor-registry-6fb5f75557.log

I tried different ways also migrate to s3 instead of file system but still same we are using helm charts Also I changed versions and tried to see are they fixed in dev or not