Open dor-hayun opened 2 months ago
I am seeing the same issue where it took almost 20 GB RSS memory
I tested with syft 1.5.0 and 1.4.1 on an Ubuntu 22.04 on an EC2 system with 32 GB of RAM and 8 vCPU and saw both of them having this issue
Test was run against a 5.5+ GB image on GCR that has Maven components on an RHEL 7.9 Image
syft ran for quite some time and then exited without any error
An initial look shows that, depending on the image being scanned, the CSV reader used within the mimetype detector lib is what's eating much of the total allocated space
It seems like we're using an older version of mimetype that does not incorporate https://github.com/gabriel-vasile/mimetype/pull/355 . When I bump the dependency and incorporate this fix though, I see the memory allocated within the mimetype.DetectReader()
call drop from 1.1GB to 740MB, which is an improvement, but I was expecting much less consumption.
I'll see what else I can do here, but since much of the consumption is from the CSV and TSV detectors alone, I'm considering dropping those detectors entirely (which would require a fork in the short term).
I've got a prototype csv/tsv detector that is pretty bare-boned, but it drops the total memory allocation from 740MB to 330MB. I'll see if I can get that PR tested and in the upstream.
@abhiseksanyal the screenshot is showing 9GB being used -- are you describing two different invocations?
syft ran for quite some time and then exited without any error
did syft display an SBOM result? Or exited without error or SBOM result?
The PR that attempts to reduce total memory allocation is stalled for a while https://github.com/anchore/mimetype/pull/2
@wagoodman thank you very much, will it be part of the next release of Syft?
Hi @wagoodman , any update here?
What happened: High memory consumption when scanning 5GB image
What you expected to happen: controlled memory consumption
Steps to reproduce the issue: scan very big sized images and check the memory consumption
Anything else we need to know?:
Environment:
syft version
: 1.5.0cat /etc/os-release
or similar): seems like there are many dotnet packages in the images