Crash when accessing large repository

[glensc]

The repository has 4020 tags, and most of them are with broken digest. I'm trying to list the tags so I could push over and delete them, but lstags crashes...

$ lstags --version
VERSION: v1.2.21
$ lstags gitlab.example.net:4567/blah/builds | awk '$2 == "this.image.is.bad.it.has.no.digest.fuuu!" {print $5}'
INFO[0000] BATCH 1 of 1
INFO[0000] ANALYZE gitlab.example.net:4567/blah/builds
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7f4238]

goroutine 10387 [running]:
github.com/ivanilves/lstags/api/v1/registry/client/request.Perform(0xc001ed7c70, 0x66, 0xc0048ce500, 0x492, 0x927fa5, 0x2, 0x400, 0x2, 0x1dcd65000, 0xc001ed7c70, ...)
        /go/src/github.com/ivanilves/lstags/api/v1/registry/client/request/request.go:116 +0x3f8
github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).v1TagOptions(0xc0002e1ce0, 0xc002046ea0, 0x22, 0xc00019c060, 0x18, 0xc00019c060, 0x18, 0xc003e68ea0)
        /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:339 +0x2a5
github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).Tag(0xc0002e1ce0, 0xc002046ea0, 0x22, 0xc00019c060, 0x18, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:381 +0x130
github.com/ivanilves/lstags/tag/remote.FetchTags.func1(0xc0002e1ce0, 0xc00021ed80, 0xc00019c060, 0x18, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/ivanilves/lstags/tag/remote/remote.go:104 +0x8b
created by github.com/ivanilves/lstags/tag/remote.FetchTags
        /go/src/github.com/ivanilves/lstags/tag/remote/remote.go:98 +0x46f

[ivanilves]

Hey @glensc

Thanks for reporting this. I haven't touched lstags for a while, but this issue looks like a good opportunity to return to action.

[daknin]

I've also run into this with approx 2500 tags

$ lstags --no-ssl-verify --concurrent-requests=2 myregistry:5000/my-image-name > my-image-name.txt
INFO[0000] BATCH 1 of 1                                 
INFO[0000] ANALYZE myregistry:5000/my-image-name 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7f3ee8]

goroutine 9114 [running]:
github.com/ivanilves/lstags/api/v1/registry/client/request.Perform(0xc003028ff0, 0x41, 0xc003b2eae0, 0x5, 0x927cc7, 0x2, 0x0, 0x2, 0x1dcd65000, 0xc003028ff0, ...)
    /go/src/github.com/ivanilves/lstags/api/v1/registry/client/request/request.go:116 +0x3f8
github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).tagDigest(0xc0002f72d0, 0x7ffc34d68ec7, 0x16, 0xc0004a8867, 0x7, 0x0, 0x459e31, 0xc0002f72d0, 0xc000220c60)
    /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:257 +0x2a5
github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).Tag.func1(0xc0002f72d0, 0x7ffc34d68ec7, 0x16, 0xc0004a8867, 0x7, 0xc003b38960, 0xc003b389c0)
    /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:349 +0x70
created by github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).Tag
    /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:348 +0xe0

and

$ cat my-image-name.txt

Will retry 'https://myregistry:5000/v2/my-image-name/manifests/0f096ac' [v2] in a 2s
=> Error: Get https://myregistry:5000/v2/my-image-name/manifests/0f096ac: dial tcp 10.11.12.13:5000: socket: too many open files
Will retry 'https://myregistry:5000/v2/my-image-name/manifests/0c1ec84' [v1] in a 2s
=> Error: Get https://myregistry:5000/v2/my-image-name/manifests/0c1ec84: dial tcp 10.11.12.13:5000: socket: too many open files
Will retry 'https://myregistry:5000/v2/my-image-name/manifests/0c1ec84' [v2] in a 2s
=> Error: Get https://myregistry:5000/v2/my-image-name/manifests/0c1ec84: dial tcp 10.11.12.13:5000: socket: too many open files
Will retry 'https://myregistry:5000/v2/my-image-name/manifests/12fb1bb' [v1] in a 2s
=> Error: Get https://myregistry:5000/v2/my-image-name/manifests/12fb1bb: dial tcp 10.11.12.13:5000: socket: too many open files
Will retry 'https://myregistry:5000/v2/my-image-name/manifests/12fb1bb' [v2] in a 2s
=> Error: Get https://myregistry:5000/v2/my-image-name/manifests/12fb1bb: dial tcp 10.11.12.13:5000: socket: too many open files
Will retry 'https://myregistry:5000/v2/my-image-name/manifests/0c1ec84' [v2] in a 4s
=> Error: Get https://myregistry:5000/v2/my-image-name/manifests/0c1ec84: dial tcp 10.11.12.13:5000: socket: too many open files
Will retry 'https://myregistry:5000/v2/my-image-name/manifests/0c1ec84' [v1] in a 4s
=> Error: Get https://myregistry:5000/v2/my-image-name/manifests/0c1ec84: dial tcp 10.11.12.13:5000: socket: too many open files
Will retry 'https://myregistry:5000/v2/my-image-name/manifests/12fb1bb' [v1] in a 4s
=> Error: Get https://myregistry:5000/v2/my-image-name/manifests/12fb1bb: dial tcp 10.11.12.13:5000: socket: too many open files
Will retry 'https://myregistry:5000/v2/my-image-name/manifests/12fb1bb' [v2] in a 4s
=> Error: Get https://myregistry:5000/v2/my-image-name/manifests/12fb1bb: dial tcp 10.11.12.13:5000: socket: too many open files

It looks like an attempt is being made to read the response after the retry attempts have been exhausted.

I wonder if the underlying issue is that connections are not being closed. Doing this let me run the failing command

$ ulimit -n
1024
$ ulimit -n 4000
$ lstags --no-ssl-verify --concurrent-requests=2 myregistry:5000/my-image-name > my-image-name.txt

[glensc]

Definitely a fd leak. I was able to avoid the nil pointer by increasing ulimit -n 4000. my repo had 1948 tags.

[glensc]

cat my-image-name.txt -- it contains error messages? the program should no print diagnostics to stdout, but to stderr. lol.

[glensc]

lstags of image with 1114 tags. crash.

$ lstags gitlab.example.net:4567/blah/builds
INFO[0000] BATCH 1 of 1
INFO[0000] ANALYZE gitlab.example.net:4567/blah/builds
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x739b06]

goroutine 11028 [running]:
github.com/ivanilves/lstags/api/v1/registry/client/request.Perform({0xc003d651a0, 0x59}, {0xc004980a00, 0x48f}, {0x7fd3fe, 0x2}, 0xf3, 0xc00020e170, 0x0)
        /go/src/github.com/ivanilves/lstags/api/v1/registry/client/request/request.go:116 +0x2e6
github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).tagDigest(0xc00032f3b0, {0xc002ed7cc0, 0x20}, {0xc00020e170, 0xd})
        /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:280 +0x1b9
github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).Tag.func1(0x1e, 0xc003124dc0)
        /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:372 +0x4d
created by github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).Tag
        /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:371 +0x196

$ lstags --version
VERSION: v1.2.23

$ ulimit -n
1024

and retry with ulimit -n 4096 passes.

[ivanilves]

Thank you for this research and for the ulimit tips @glensc :bow:

Unluckily I was not able to dedicate all the planned time to lstags project this year. I would really like to have it patched, to at least avoid fd leak issues and stdout/stderr misuse, however I can not make any exact time/effort promises :innocent: