mtrmac
commented
2 years ago Thanks for your report.
Yes, this does look somewhat similar to https://github.com/containers/image/issues/1083 , and the cause is unknown. With EOF (and not unexpected EOF) it seems like a code bug somewhere, whether or not there is some underlying infrastructure issue ultimately causing the transfer to fail.
If you can reproduce this on-demand, that’s valuable.
- Is there anything interesting in the registry logs?
- Is there some indication that this is related to a possibly configurable timeout? E.g. if the log entries are instrumented with timestamps, there should be a
PATCH $urlentry corresponding to thePatch $urlerror message fragment (curiously that’s missing in the above log). Is the time difference between thePATCHlog entry and the ultimate failure close to some suspiciously round number like 30/60/… seconds? - It might be possible to observe the network traffic using standard tools like Wireshark (decrypting TLS?) to see if there is anything clearly unusual happening.
- From a code perspective, ultimately it should be possible to debug this just by patching the error paths to identify which one was taken; that requires some Go knowledge, but otherwise it’s mostly just mechanical work. (E.g. start with
vendor/github.com/containers/image/v5/docker/docker_client.go:makeRequestToResolvedURL makeRequestToResolvedURLand on everyreturnpath, add a log entry to record which return path was used, and with what error value. E.g. something likefunc (c *dockerClient) makeRequestToResolvedURL(ctx context.Context, method string, url *url.URL, headers map[string][]string, stream io.Reader, streamLen int64, auth sendAuth, extraScope *authScope) (*http.Response, error) { … logrus.Errorf("makeRequestToResolvedURL RETURN 1: returning %#v, %#v; context %#v, %#v", res, err, stream, attempts) return res, err … logrus.Errorf("makeRequestToResolvedURL RETURN 2: returning %#v, ctx.Err()) return nil, ctx.Err()and the like. After that identifies where the code failed, that could either be enough to identify the cause directly, or it will identify a function that failed (in this case, it’s very likely (but not, strictly speaking, 100% certain) that it’s inside
makeRequestToResolvedURL, probably innet/http.Client.Do)), and that one can be traced in a similar way. It’s likely that the standard library in/usr/local/gowill need (temporarily!) instrumenting the same way…
yuyanj143
github-actions[bot]
Hi all, I'm trying to copy a fairly large image (10GB) from my local OCI-layout directory to my private docker registry hosted on s3 using:
skopeo --debug --insecure-policy copy --dest-creds=user.momo:pwd oci:/home/momo/oci:new docker://my-private-registrybut this is consistently failing with EOF errors:
Additional info: skopeo version: 1.3.0 docker-registry version: 2.7.1 cat /etc/*release
registry config:
I've tried the following and they work:
I suspect this is somewhat similar to issue 1083 but mine doesn't involve a slow/fast pipe as I'm copying from the local directory. I'm looking for some tips how to debug and if there's any workaround (I'm open to patch skopeo) for pushing large images with skopeo. Thanks!