search

containerd / nerdctl

contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ...
Apache License 2.0
7.86k stars 585 forks source link

TestTailFollowRotateLogs is still failing #3225

Closed apostasie closed 4 weeks ago

apostasie commented 1 month ago

Description

Despite many tries to tweak the timeout, log length or log parameters, this test still fails regularly.

Even locally, this:

while true; do go test ./cmd/nerdctl -count 1 -run TestTailFollowRotateLogs; done

Will fail at least once in ten times.

I am now thinking there might actually be an underlying bug.

Could it be the same issue as #3199 ?

Log specialists, any idea? (@fahedouch maybe?)

Steps to reproduce the issue

na

Describe the results you received and expected

na

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

apostasie commented 1 month ago

Note: after testing against PR #3200 the issue is still there.

xyz-li commented 1 month ago

https://github.com/containerd/nerdctl/blob/bec95c4e43aa70511fca7189dd9b9f4fe4261461/cmd/nerdctl/container_logs_test.go#L224-L229

I think max-size is too small and log output is too quicklly. There is one situation, startTail return with true, and nerdctl rotate the logfile, then openFileShareDelete will return an error. At last this test will fail.

How about change linesPerFile to 1000.

root@lh:/root/go/src/github.com/containerd/nerdctl# while true; do go test ./cmd/nerdctl -count 1 -run TestTailFollowRotateLogs; done
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.168s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.688s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.447s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    3.688s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.488s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.658s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.904s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    2.362s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.581s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.527s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.124s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.902s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.020s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.410s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.846s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.540s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    2.973s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.168s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.679s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.667s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.335s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.582s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.449s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.467s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    2.151s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    2.055s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.971s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.490s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.785s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.126s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.863s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    2.637s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.546s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.115s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.578s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.573s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.567s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    3.680s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    4.220s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.636s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.087s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    5.068s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    3.755s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.726s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.672s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.745s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.535s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.689s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.542s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    2.160s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.372s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.938s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.543s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.634s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.220s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    2.223s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.270s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.510s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.615s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    3.755s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.567s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.673s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.727s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.399s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.160s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.675s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.667s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    2.495s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    4.484s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.106s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.194s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    0.612s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    1.059s
apostasie commented 1 month ago

How about change linesPerFile to 1000.

@xyz-li anything to make this test stop failing of course :-)

Do you think there is an underlying issue we can look into though?

xyz-li commented 1 month ago

I think max-size is too small and log output is too quicklly. There is one situation, startTail(or os.OpenFile) return with true, and nerdctl rotate the logfile, then openFileShareDelete will return an error. At last this test will fail.

After change linesPerFile to 2000. 

# while true; do go test ./cmd/nerdctl -count 1 -run TestTailFollowRotateLogs; done
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.327s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.355s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.337s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.349s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.487s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.404s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.360s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.275s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    13.349s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.660s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.700s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.613s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.335s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.574s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.307s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.373s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.491s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.445s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.464s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.497s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.739s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.517s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.321s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.399s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.396s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.423s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.558s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.411s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.406s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.393s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.385s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.415s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.439s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.446s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.690s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.456s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.411s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.483s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.416s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.422s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.421s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.291s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.334s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.357s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.461s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.412s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.440s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.505s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.519s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.276s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.427s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.369s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.484s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.440s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.437s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.397s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.384s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.362s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.649s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.787s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.520s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.410s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.432s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.381s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.158s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.953s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.980s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.473s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.412s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.251s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.419s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.795s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.522s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.618s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.277s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.356s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.178s
ok      github.com/containerd/nerdctl/v2/cmd/nerdctl    12.406s