search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.15k stars 1.16k forks source link

Took more time on a simple step than usual #4575

Open hezhizhen opened 9 months ago

hezhizhen commented 9 months ago

There's a simple step in my Dockerfile:

RUN mv /usr/local/jdk1.8.0 /usr/local/jdk1.8.0.old

And it usually needs about 1 second:

[2024-01-23 14:12:34.121]  #16 [ 3/23] RUN mv /usr/local/jdk1.8.0 /usr/local/jdk1.8.0.old
[2024-01-23 14:12:34.872]  #16 DONE 0.8s
[2024-01-23 12:51:47.603]  #16 [ 3/23] RUN mv /usr/local/jdk1.8.0 /usr/local/jdk1.8.0.old
[2024-01-23 12:51:48.805]  #16 DONE 1.1s

But sometimes it takes much more time:

[2024-01-16 14:54:20.487]  #16 [ 3/23] RUN mv /usr/local/jdk1.8.0 /usr/local/jdk1.8.0.old
[2024-01-16 14:54:37.154]  #16 DONE 16.8s

The buildkitd version is v0.12.0 and the buildctl version is v0.12.1.

jedevc commented 9 months ago

@hezhizhen thanks for the report :tada:

Could you look through https://github.com/moby/buildkit/blob/master/.github/issue_reporting_guide.md, and share the contents of your buildkitd logs (or the docker daemon logs, if running through buildkit-in-docker). Additionally, it would help if you could describe the environment that you're running this on.

hezhizhen commented 9 months ago

Here's buildkitd's log at that time:

Jan 16 14:51:49 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:51:49+08:00" level=warning msg="skip to append distribution label: label key and value length (4130 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=b69558591900b11f traceID=0c6b192d8f4635f91642f6f8e44ede65
Jan 16 14:51:49 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:51:49+08:00" level=warning msg="skip to append distribution label: label key and value length (4131 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=b69558591900b11f traceID=0c6b192d8f4635f91642f6f8e44ede65
Jan 16 14:51:49 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:51:49+08:00" level=warning msg="skip to append distribution label: label key and value length (4129 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=b69558591900b11f traceID=0c6b192d8f4635f91642f6f8e44ede65
Jan 16 14:51:57 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:51:57+08:00" level=warning msg="skip to append distribution label: label key and value length (4106 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=dc16c2e38fc5601e traceID=e903bb5f464d7654c510c22e83a7f629
Jan 16 14:51:57 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:51:57+08:00" level=warning msg="skip to append distribution label: label key and value length (4106 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=dc16c2e38fc5601e traceID=e903bb5f464d7654c510c22e83a7f629
Jan 16 14:51:57 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:51:57+08:00" level=warning msg="skip to append distribution label: label key and value length (4106 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=dc16c2e38fc5601e traceID=e903bb5f464d7654c510c22e83a7f629
Jan 16 14:51:57 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:51:57+08:00" level=warning msg="skip to append distribution label: label key and value length (4106 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=dc16c2e38fc5601e traceID=e903bb5f464d7654c510c22e83a7f629
Jan 16 14:51:57 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:51:57+08:00" level=warning msg="skip to append distribution label: label key and value length (4114 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=dc16c2e38fc5601e traceID=e903bb5f464d7654c510c22e83a7f629
Jan 16 14:54:40 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:54:40+08:00" level=warning msg="skip to append distribution label: label key and value length (4111 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=ed7671facc4b2c3f traceID=048c74682e99b737516cd75b446d65a5
Jan 16 14:54:40 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:54:40+08:00" level=warning msg="skip to append distribution label: label key and value length (4111 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=ed7671facc4b2c3f traceID=048c74682e99b737516cd75b446d65a5
Jan 16 14:55:00 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:55:00+08:00" level=warning msg="skip to append distribution label: label key and value length (4112 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=14847d14b427b53d traceID=311c15ee21c374620ed6fc36997dc92e
Jan 16 14:55:00 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:55:00+08:00" level=warning msg="skip to append distribution label: label key and value length (4104 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=14847d14b427b53d traceID=311c15ee21c374620ed6fc36997dc92e
Jan 16 14:55:00 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:55:00+08:00" level=warning msg="skip to append distribution label: label key and value length (4115 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=14847d14b427b53d traceID=311c15ee21c374620ed6fc36997dc92e
Jan 16 14:55:00 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:55:00+08:00" level=warning msg="skip to append distribution label: label key and value length (4115 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=14847d14b427b53d traceID=311c15ee21c374620ed6fc36997dc92e
Jan 16 14:55:00 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:55:00+08:00" level=warning msg="skip to append distribution label: label key and value length (4104 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=14847d14b427b53d traceID=311c15ee21c374620ed6fc36997dc92e
Jan 16 14:55:00 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:55:00+08:00" level=warning msg="skip to append distribution label: label key and value length (4115 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=14847d14b427b53d traceID=311c15ee21c374620ed6fc36997dc92e
Jan 16 14:55:00 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:55:00+08:00" level=warning msg="skip to append distribution label: label key and value length (4115 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=14847d14b427b53d traceID=311c15ee21c374620ed6fc36997dc92e
Jan 16 14:55:01 jscs-nyx-node-8 buildkitd[2279553]: time="2024-01-16T14:55:01+08:00" level=warning msg="skip to append distribution label: label key and value length (4114 bytes) greater than maximum size (4096 bytes), key: containerd.io/distribution.source.hub.xxxx.co: invalid argument" spanID=14847d14b427b53d traceID=311c15ee21c374620ed6fc36997dc92e

The buildkitd is deployed on the host and run with systemd.

jedevc commented 9 months ago

Can you start buildkitd with --debug to enable additional logging? Any additional context here would be helpful, unfortunately, those logs don't really reveal much :cry:

hezhizhen commented 9 months ago

Okay

tonistiigi commented 9 months ago

Maybe also try running RUN strace mv ... and report the output.

hezhizhen commented 9 months ago

Maybe also try running RUN strace mv ... and report the output.

Thanks for your advice and I'll try it next time I encounter it.