search

goharbor / harbor-helm

The helm chart to deploy Harbor
Apache License 2.0
1.18k stars 758 forks source link

Image push fails with retries and never completes #1445

Closed natarajmb closed 1 year ago

natarajmb commented 1 year ago

I have installed this using Helm Chart (now have tried both with Harbor and Bitnami charts) still the same issue.

cluster config RPI cluster running on microk8s with openebs hostpath + nginx ingress controller + metallb

usecase client -> haproxy (ssl passthrough) -> nginx Ingress (ssl passthrough) -> Harbor (ssl) image I'm uploading is about 300mb

$ docker push registry.xxx.xxx.dev/library/tools-test
Using default tag: latest
The push refers to repository [registry.xxx.xxx.dev/library/tools-test]
3a0cbea8ddd6: Pushing [==================================================>]  10.08MB
5dff9cf2a862: Pushing [==================================================>]  66.11MB/66.11MB
d7da46359a31: Pushing [===============================================>   ]  44.73MB/46.92MB
2a43af9c4ee6: Pushing [==================================================>]  50.86MB/50.86MB
6515074984c6: Pushing [==================================================>]   77.8MB/77.8MB
EOF

I have looked at the logs on both ingress and harbor-core and seeing this and later just to be sure whats happening watched the traffic with kubeshark (attached is a screenshot)

2023/03/12 17:39:24 http: proxy error: context canceled
2023/03/12 17:39:24 http: proxy error: context canceled
2023/03/12 17:39:24 http: proxy error: context canceled
2023/03/12 17:39:26 http: proxy error: context canceled
Screenshot 2023-03-12 at 17 47 51

I have this annotation on the ingress

nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
nginx.ingress.kubernetes.io/server-snippet: "keepalive_timeout 3600s;client_body_timeout 3600s;"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"

I see the client doesn't get an acknowledgement back and keeps retrying and re-uploading. I have tried for a few days now and lost at ends hence posting it here for help.

natarajmb commented 1 year ago

If anyone still looking at this and maybe for future reference to self. The issue is the way I have my network configured. I have exposed the harbor service externally and also have an internal DNS resolver entry for the internal network. Trying to push images from outside works. It was failing for pushing images through the internal network. Removing the DNS resolver entry for the internal network (DNS Resolver entry on pfsense) fixed the issue. The drawback being someone pushing images from the internal network still takes the external path, which I can live with as an internal user will mostly be me.