search

containerd / nydus-snapshotter

A containerd snapshotter with data deduplication and lazy loading in P2P fashion
https://nydus.dev/
Apache License 2.0
168 stars 96 forks source link

CI: randomly failed e2e test and continuous prefetch warnings #555

Open hangvane opened 11 months ago

hangvane commented 11 months ago

https://github.com/containerd/nydus-snapshotter/actions/runs/6739821947/job/18321972302

time="2023-11-03T00:41:46Z" level=fatal msg="connection error: desc = \"transport: Error while dialing dial unix:///run/containerd-nydus/containerd-nydus-grpc.sock: timeout\": unavailable"
[2023-11-03 00:41:46.650093 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob e4d6efbd0a0d130541b68d7fe421474184657b37eff8fef1d6e3048454b0c671, offset 0, size 6424, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-11-03 00:41:46.650321 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob b194a89b4d14dfe137e08bc984b7491b079e169597fab03c6e92d1c7e76fbc97, offset 0, size 5697, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-11-03 00:41:46.650374 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 35abf43dce465d40f45a4445af0c570dcd4dc36faa6a225928aeb64c122c3a08, offset 10485760, size 146731, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-11-03 00:41:46.651170 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob a8f84a5d40140aa0586b41668a28f9579d0a6c9dc0fdc8abb09197cef00a2cef, offset 0, size 9415, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend

Seems that the prefetch warnings in e2e test exist for at least 2 months, and may randomly fail e2e test.

Additionally, the warning infomation is covered by Invalid argument (os error 22), making it hard to trace the error.

imeoer commented 11 months ago

It seems this error happened earlier:

time="2023-11-03T00:41:42.464732357Z" level=warning msg="snapshot garbage collection failed" error="error reading from server: EOF: unavailable" snapshotter=nydus

It looks like the nydusd HTTP API server is broken.