goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.2k stars 4.77k forks source link

nginx-python send RST package to client cause EOF #20965

Open suchen-sci opened 1 month ago

suchen-sci commented 1 month ago

Hi Team,

We are currently running the Harbor project using Docker Compose. When accessing Nginx-Python with our Golang client, we have encountered an issue where, approximately every few hours, the client experiences an EOF error. Upon investigating with tcpdump, we observed that about one minute after the request, Nginx-Python sometimes sends a RST packet to the client, which results in the Golang client receiving an EOF.

Could you please provide any advice or insights on how to resolve this issue?

Below are the settings for our Golang client:

func HTTPClient(tlsCfg *tls.Config) *http.Client {
    dialFunc := func(ctx stdctx.Context, network, addr string) (net.Conn, error) {
        return (&net.Dialer{
            Timeout:   30 * time.Second,
            KeepAlive: 60 * time.Second,
        }).DialContext(ctx, network, addr)
    }

    client := &http.Client{
        Transport: &http.Transport{
            Proxy:              http.ProxyFromEnvironment,
            DialContext:        dialFunc,
            TLSClientConfig:    tlsCfg,
            DisableCompression: false,
            // NOTE: The large number of Idle Connections can
            // reduce overhead of building connections.
            IdleConnTimeout:       90 * time.Second,
            TLSHandshakeTimeout:   10 * time.Second,
            ExpectContinueTimeout: 1 * time.Second,
        },
    }
    return client
}

Thanks a lot!

wy65701436 commented 1 month ago

Hi, providing more details will help us better understand your issue.

suchen-sci commented 1 month ago

Hi, team

Thanks a lot for your replay. I am so sorry for the delay of response.

  1. We use harbor v2.10.3 for all containers.
  2. We use /api/v2.0/health to obtain health status of harbor.
  3. There is not non-2XX response in proxy logs. the logs of health look like this xx.xx.xx.xx - "GET /api/v2.0/health HTTP/1.1" 200 305 "-" "Go-http-client/1.1" 0.002 0.002.

For more information, we get this error when use docker resolver to push layer to harbor service.

time="2024-09-29T10:08:11.441934605Z" level=error msg="failed to build overlaybd: failed to build \"sha256:xxx\": failed to build /tmp/overlaybd/blobs/139: failed to upload layer 20: failed to upload layer 20: failed to copy: failed to do request: Put \"https://harbor-ip/v2/user-123/model-123/blobs/uploads/xxx": write tcp localhost-ip:59480->harbor-ip:20443: use of closed network connection"
MinerYang commented 1 month ago

Have you set any keep-alive settings for your http client? Or could you check if there's any improper close of your connections?