search

GoogleContainerTools / container-structure-test

validate the structure of your container images
Apache License 2.0
2.35k stars 200 forks source link

High memory usage when running tests #237

Open brianelliott-wt opened 5 years ago

brianelliott-wt commented 5 years ago

I have a 6GB docker image with a fair amount of tools installed and I am seeing some very high memory usage when running tests on it. I am seeing the Container Structure Test process take upwards of 5-6GB of memory at peak and the docker process also taking up 6-7GB of memory at its peak. The testing software does eventually clean up, but the docker process does not until its restarted.

I have a few other images around 2GB and 4GB in size and neither of them have this issue. I am running 1 file existence test and about 6 command tests that are just running a which command on various software.

Any thoughts on what is going on here? This issue is stopping me from running this in our dev pipelines since our agents can't handle that kind of memory load.

brianelliott-wt commented 5 years ago

Figured out what was going on. I probably should've assumed that checking the existence of a directory wouldn't work as well as I was hopping with a "File Existence" check.

After digging deeper, It seems that this is pulling the files specified off of the container and then checking their permissions https://github.com/GoogleContainerTools/container-structure-test/blob/5ca6fe610afaa34c8d46f550765e254bcce8ec96/pkg/drivers/docker_driver.go#L205

I feel like there could be a way to handle directories existing without doing this, but for now I've just linked to a file in the directory I am checking instead of the directory itself.

For any future people coming upon this, don't check for existence of large folder with the FileExistenceCheck or you will probably have a bad time :) It eats up a lot of memory and the docker process never cleans that up correctly.