google / go-containerregistry

Go library and CLIs for working with container registries
Apache License 2.0
3.1k stars 540 forks source link

PATCH: read {path}: file already closed error following update of go-containerregistry to v0.15.2 #2019

Open parvathikp100 opened 1 day ago

parvathikp100 commented 1 day ago

Describe the bug

We recently upgraded our go-containerregistry version from v0.11.0 to v0.15.2 and there has been a sudden surge in the number of the following

Patch "https://xxx/blobs/uploads/xxx?_state=xxx%3D%3D": read /xxx/oci/blobs/sha256/xxx: file already closed

error which seem to be originating from the http.PATCH request in streamBlob function

Also interesting is to note that similar surge in errors have been seen in http requests being made from commitManifest and initiateUpload

The major change that is common to streamBlob, initiateUpload and commitManifest is the addition of retry.Never(ctx) which seems to be affecting the retries somehow?

This is the PR that introduced it - https://github.com/google/go-containerregistry/pull/1502#issue-1480177398

To Reproduce

No clue on reproducing it yet

Expected behavior

Blobs are written successfully without errors at any of the http requests

Additional context

Added a comment on the original PR with some more details - https://github.com/google/go-containerregistry/pull/1502#issuecomment-2396940335

parvathikp100 commented 1 day ago

I also added a comment on this PR that introduced the retry predicate - https://github.com/google/go-containerregistry/pull/1502#issuecomment-2396940335.

parvathikp100 commented 1 day ago

Also see a rise in errors for other http requests originating from uploadOne():

Put "https://xxx/manifests/sha256:xxx": http: ContentLength=2706 with Body length 0
PUT https://xxx/blobs/uploads/xxx?_state=REDACTED&digest=sha256%3Axxx: DIGEST_INVALID: provided digest did not match uploaded content; map[Digest:sha256:xxx Reason:map[]]