Open Satoshi-Hirai371 opened 3 months ago
I experience a very similar issue with a 712MB image, pushing from a (Debian 12.6) Linux client with Docker 27.0.3 and Harbor v2.12.0-f86f1ceb. Smaller images work fine, but this one keeps failing.
The docker client hangs at ~400-600 MB and after timeout returns unknown: Client Closed Request
. The docker service logs:
Jul 08 08:11:22 gandalf dockerd[481773]: time="2024-07-08T08:11:22.156010705+02:00" level=error msg="Upload failed, retrying: unknown: Client Closed Request"
Jul 08 08:12:27 gandalf dockerd[481773]: time="2024-07-08T08:12:27.330959239+02:00" level=error msg="Upload failed, retrying: unknown: Client Closed Request"
Jul 08 08:13:37 gandalf dockerd[481773]: time="2024-07-08T08:13:37.633370636+02:00" level=error msg="Upload failed, retrying: unknown: Client Closed Request"
Jul 08 08:14:52 gandalf dockerd[481773]: time="2024-07-08T08:14:52.895728642+02:00" level=error msg="Upload failed, retrying: unknown: Client Closed Request"
Jul 08 08:16:13 gandalf dockerd[481773]: time="2024-07-08T08:16:13.150553594+02:00" level=error msg="Upload failed: unknown: Client Closed Request"
Jul 08 08:16:13 gandalf dockerd[481773]: time="2024-07-08T08:16:13.150661807+02:00" level=error msg="Not continuing with push after error: unknown: Client Closed Request"
The harbor.core log:
2024/07/08 08:11:22 http: proxy error: context canceled
2024/07/08 08:12:27 http: proxy error: context canceled
2024/07/08 08:13:37 http: proxy error: context canceled
2024/07/08 08:14:52 http: proxy error: context canceled
2024/07/08 08:16:13 http: proxy error: context canceled
I run Harbor on Kubernetes 1.30.1 behind a Traefik 3.0 proxy that handles LE certificates and routing. The Traefik logs show no error or warning.
@Satoshi-Hirai371 what size of your image? How did you install harbor? Through offline installer or helm chart?
I tested it from a small Linux server first and now from my M2 Pro MacBook and that works (is much faster, so no timeout issues probably).
@Satoshi-Hirai371 what size of your image? How did you install harbor? Through offline installer or helm chart?
The image is 4GIB and uses an offline installer.
This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.
Expected behavior and actual behavior: Sometimes pushing an image from a Docker client to a harbor fails. This operation is expected to push normally, but can you identify the cause from the command output and log shown below?
Steps to reproduce the problem: It may fail when pushing images of several GiB.
Versions: Please specify the versions of following systems.
Additional context:
In the Docker CLI, the push is retried once and then failed.
The Docker service was experiencing Timeout with the following message.