bug(image): INTERNAL_ERROR when scanning an image

[DmitriyLewen]

Some scanning an image return stream error: stream ID XX; INTERNAL_ERROR; received from peer error. This is not a stable bug. Only one of the few scans of the same image returns an error.

Looks like it happans only for remote images.

Related discussions:

• 5420

• 5345

[DmitriyLewen]

Hello @AshkanRafiee, @d-mankowski-synerise, @LesSyner

I can't reproduce your INTERNAL_ERROR error.

Can you help me with investigation?

I found that google-cloud-go also gets this error, and fter updating to go 1.20 they getting more errors - https://github.com/googleapis/google-cloud-go/issues/7440 If I haven't missed anything, we close Reader of the layer, as recommended. But maybe there is some conflict with go-containerregistry (it uses go 1.18).

Can you try scanning your images using Trivy v0.44.1? (we updated go to 1.20 in v0.45.0).

[LesSyner]

I tried to reproduce this error but had no luck this time. Maybe OS was also some factor and in the mean time I've upgraded macOS to 14.1. I will not judge internals but my impression was that it's some issue with internal cache when image is downloaded by trivy, when I had this issue and downloaded image to local docker it stopped immediately. On the other hand I still have similar error with exactly this image when it's scanned with trivy-operator, which has trivy 0.45.1 inside. I know it's different setup but anyway trivy 0.45.1 with go-containerregistry 1.20. We scan over 300 images, consistently 6 of them have this or similar issue and debezium/connect:1.9.0.Final is one of them (other are internal ones so harder for you to investigate).

[AshkanRafiee]

Hello, @DmitriyLewen

Upon your request, I have conducted tests using Trivy versions 0.44.1, 0.45.0, and 0.46.1 with official dockerized versions. I aimed to replicate the INTERNAL_ERROR issue by scanning both remotely and locally, utilizing the vulnerability and secret detection features. Before each test, I cleared the cache to ensure a clean state.

I am surprised to report that the error did not recur in any of the scenarios tested. This leads me to consider the possibility that the discrepancy in our experiences might be due to differences in the operating system environments. An additional variable might be the method of Trivy installation – in my case, it was installed via Homebrew, which could have influenced the outcome somehow. Furthermore, other variables such as database updates or other changes related to the OS might also play a role in this behavior.

It's really interesting that @LesSyner has also not been able to reproduce the error, which aligns these findings. Currently, I'm running Trivy on Windows Subsystem for Linux (WSL) with an Ubuntu distribution. Given these observations, it might be worth considering environment-specific factors that could affect the occurrence of the error.

[LesSyner]

To add to things described by @AshkanRafiee - previously I've used trivy installed by homebrew and it was on macOS 13.x. Now my tests with trivy 0.44.1 were done using binary downloaded from trivy releases page (since homewbrew has latest version) and it was on macOS 14.1. Errors which I see now (and they were also during my initiial tests) are for trivy 0.45.1 running in server mode on k8s cluster and utilised by trivy-operator 0.16.3.

[DmitriyLewen]

Thanks for your investigations.

I have 1 more idea - maybe it has to do with the version of Go used to build Trivy, because there are cases when people see this error when trying to download Go packages - https://github.com/golang/go/issues/51323

trivy-operator already updated Trivy to v0.47.0 - https://github.com/aquasecurity/trivy-operator/blob/089856b4f8d5705f997303884f6444f075b07d32/go.mod#L8

Can you try to reproduce this error when new version of trivy-operator will be released, please?

[LesSyner]

Sure I'll check it once they release new version with trivy 0.47 included. Let's have fingers crossed.