search

anchore / stereoscope

go library for processing container images and simulating a squash filesystem
Apache License 2.0
78 stars 43 forks source link

More memory efficient mimetype detection #267

Closed wagoodman closed 2 months ago

wagoodman commented 2 months ago

This PR incorporates https://github.com/gabriel-vasile/mimetype/pull/553 via the temporary fork. Before this improvement this was the relative performance for a sample image via syft:

Screenshot 2024-07-10 at 1 13 44 PM

Specifically the mimetype detection was allocating 1.1GB of memory:

Screenshot 2024-07-10 at 1 14 13 PM

With the upstream change incorporated we're using ~800 less MB of memory (reduced 1.1GB to ~340MB) to do the same work:

Screenshot 2024-07-10 at 1 13 56 PM Screenshot 2024-07-10 at 1 14 26 PM

Note the figures are for total allocated space and not inuse memory.

Partially addresses #233

github-actions[bot] commented 2 months ago

Benchmark Test Results

Benchmark results from the latest changes vs base branch ``` make .tool/task make[1]: Entering directory '/home/runner/work/stereoscope/stereoscope' make[1]: Leaving directory '/home/runner/work/stereoscope/stereoscope' .tool/task show-benchstat ? github.com/anchore/stereoscope [no test files] ? github.com/anchore/stereoscope/examples [no test files] PASS ok github.com/anchore/stereoscope/internal 0.003s ? github.com/anchore/stereoscope/internal/bus [no test files] PASS ok github.com/anchore/stereoscope/internal/containerd 0.008s PASS ok github.com/anchore/stereoscope/internal/docker 0.005s ? github.com/anchore/stereoscope/internal/log [no test files] PASS ok github.com/anchore/stereoscope/internal/podman 0.005s ? github.com/anchore/stereoscope/pkg/event [no test files] ? github.com/anchore/stereoscope/pkg/event/parsers [no test files] goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor BenchmarkTarIndex-4 33375 35777 ns/op 5699 B/op 93 allocs/op BenchmarkTarIndex-4 33187 35743 ns/op 5699 B/op 93 allocs/op BenchmarkTarIndex-4 33516 35828 ns/op 5703 B/op 93 allocs/op BenchmarkTarIndex-4 33344 35889 ns/op 5699 B/op 93 allocs/op BenchmarkTarIndex-4 33582 35737 ns/op 5698 B/op 93 allocs/op BenchmarkTarIndex-4 33516 35827 ns/op 5698 B/op 93 allocs/op BenchmarkTarIndex-4 33442 36007 ns/op 5700 B/op 93 allocs/op PASS ok github.com/anchore/stereoscope/pkg/file 10.942s PASS ok github.com/anchore/stereoscope/pkg/filetree 0.005s ? github.com/anchore/stereoscope/pkg/filetree/filenode [no test files] PASS ok github.com/anchore/stereoscope/pkg/image 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/containerd 0.009s PASS ok github.com/anchore/stereoscope/pkg/image/docker 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/oci 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/oci/credhelpers 0.006s ? github.com/anchore/stereoscope/pkg/image/podman [no test files] PASS ok github.com/anchore/stereoscope/pkg/image/sif 0.005s ? github.com/anchore/stereoscope/pkg/imagetest [no test files] PASS ok github.com/anchore/stereoscope/pkg/tree 0.003s PASS ok github.com/anchore/stereoscope/pkg/tree/node 0.004s goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/test/integration cpu: AMD EPYC 7763 64-Core Processor BenchmarkSimpleImage_GetImage/docker-archive-4 987 1208382 ns/op 290047 B/op 2472 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 990 1215180 ns/op 289720 B/op 2472 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 987 1246097 ns/op 289678 B/op 2471 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 991 1228771 ns/op 289523 B/op 2471 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 988 1195063 ns/op 289329 B/op 2471 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 994 1270525 ns/op 289369 B/op 2471 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 985 1279316 ns/op 289476 B/op 2471 allocs/op BenchmarkSimpleImage_GetImage/podman-4 63 18281529 ns/op 410106 B/op 2676 allocs/op BenchmarkSimpleImage_GetImage/podman-4 58 19324548 ns/op 409704 B/op 2676 allocs/op BenchmarkSimpleImage_GetImage/podman-4 63 19146959 ns/op 410186 B/op 2676 allocs/op BenchmarkSimpleImage_GetImage/podman-4 61 19139812 ns/op 409459 B/op 2674 allocs/op BenchmarkSimpleImage_GetImage/podman-4 62 18680640 ns/op 409708 B/op 2675 allocs/op BenchmarkSimpleImage_GetImage/podman-4 64 18809938 ns/op 409431 B/op 2675 allocs/op BenchmarkSimpleImage_GetImage/podman-4 64 18849104 ns/op 409408 B/op 2676 allocs/op #0 building with "default" instance using docker driver #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 345B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 transferring context: 2B done #2 DONE 0.0s #3 [internal] load build context #3 transferring context: 209B done #3 DONE 0.0s #4 [1/3] ADD file-1.txt /somefile-1.txt #4 CACHED #5 [2/3] ADD file-2.txt /somefile-2.txt #5 CACHED #6 [3/3] ADD target / #6 CACHED #7 exporting to image #7 exporting layers done #7 writing image sha256:fd49a7e75819d889d09a10c48dfa50867cba5b0e5620be56c53a7b708ed631ce done #7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done #7 DONE 0.0s ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" --- FAIL: BenchmarkSimpleImage_GetImage image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2024-07-10 17:23:50.048392572 +0000 UTC, mode: -rw-r--r--) image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7" image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 image_fixtures.go:286: Error Trace: /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33 /home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:163 Error: Received unexpected error: exit status 1 Test: BenchmarkSimpleImage_GetImage Messages: could not import docker image to containerd (shell out) BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53836 22263 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53839 22129 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53907 22145 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54104 22112 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54279 22106 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54148 22055 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53942 22068 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54199 22043 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54394 22055 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54304 22115 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53862 22155 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54034 22006 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54447 22046 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54042 21974 ns/op 2712 B/op 21 allocs/op #0 building with "default" instance using docker driver #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 345B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 transferring context: 2B done #2 DONE 0.0s #3 [internal] load build context #3 transferring context: 209B done #3 DONE 0.0s #4 [1/3] ADD file-1.txt /somefile-1.txt #4 CACHED #5 [2/3] ADD file-2.txt /somefile-2.txt #5 CACHED #6 [3/3] ADD target / #6 CACHED #7 exporting to image #7 exporting layers done #7 writing image sha256:fd49a7e75819d889d09a10c48dfa50867cba5b0e5620be56c53a7b708ed631ce done #7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done #7 DONE 0.0s ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" --- FAIL: BenchmarkSimpleImage_FetchSquashedContents image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2024-07-10 17:23:50.048392572 +0000 UTC, mode: -rw-r--r--) image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7" image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 image_fixtures.go:286: Error Trace: /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:64 /home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:189 Error: Received unexpected error: exit status 1 Test: BenchmarkSimpleImage_FetchSquashedContents Messages: could not import docker image to containerd (shell out) FAIL exit status 1 FAIL github.com/anchore/stereoscope/test/integration 40.181s ? github.com/anchore/stereoscope/test/integration/test-fixtures/registry [no test files] FAIL goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor ctr: │ .tmp/benchmark-fb32042.txt │ │ sec/op │ TarIndex-4 35.83µ ± 1% │ .tmp/benchmark-fb32042.txt │ │ B/op │ TarIndex-4 5.565Ki ± 0% │ .tmp/benchmark-fb32042.txt │ │ allocs/op │ TarIndex-4 93.00 ± 0% pkg: github.com/anchore/stereoscope/test/integration │ .tmp/benchmark-fb32042.txt │ │ sec/op │ SimpleImage_GetImage/docker-archive-4 1.229m ± 4% SimpleImage_GetImage/podman-4 18.85m ± 3% geomean 4.813m │ .tmp/benchmark-fb32042.txt │ │ B/op │ SimpleImage_GetImage/docker-archive-4 282.7Ki ± 0% SimpleImage_GetImage/podman-4 400.1Ki ± 0% geomean 336.3Ki │ .tmp/benchmark-fb32042.txt │ │ allocs/op │ SimpleImage_GetImage/docker-archive-4 2.471k ± 0% SimpleImage_GetImage/podman-4 2.676k ± 0% geomean 2.571k ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" │ .tmp/benchmark-fb32042.txt │ │ sec/op │ SimpleImage_FetchSquashedContents/docker-archive-4 22.11µ ± 1% SimpleImage_FetchSquashedContents/podman-4 22.05µ ± 0% geomean 22.08µ │ .tmp/benchmark-fb32042.txt │ │ B/op │ SimpleImage_FetchSquashedContents/docker-archive-4 2.648Ki ± 0% SimpleImage_FetchSquashedContents/podman-4 2.648Ki ± 0% geomean 2.648Ki │ .tmp/benchmark-fb32042.txt │ │ allocs/op │ SimpleImage_FetchSquashedContents/docker-archive-4 21.00 ± 0% SimpleImage_FetchSquashedContents/podman-4 21.00 ± 0% geomean 21.00 goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor ctr: │ .tmp/benchmark-fb32042.txt │ │ sec/op │ TarIndex-4 35.83µ ± 1% │ .tmp/benchmark-fb32042.txt │ │ B/op │ TarIndex-4 5.565Ki ± 0% │ .tmp/benchmark-fb32042.txt │ │ allocs/op │ TarIndex-4 93.00 ± 0% pkg: github.com/anchore/stereoscope/test/integration │ .tmp/benchmark-fb32042.txt │ │ sec/op │ SimpleImage_GetImage/docker-archive-4 1.229m ± 4% SimpleImage_GetImage/podman-4 18.85m ± 3% geomean 4.813m │ .tmp/benchmark-fb32042.txt │ │ B/op │ SimpleImage_GetImage/docker-archive-4 282.7Ki ± 0% SimpleImage_GetImage/podman-4 400.1Ki ± 0% geomean 336.3Ki │ .tmp/benchmark-fb32042.txt │ │ allocs/op │ SimpleImage_GetImage/docker-archive-4 2.471k ± 0% SimpleImage_GetImage/podman-4 2.676k ± 0% geomean 2.571k ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" │ .tmp/benchmark-fb32042.txt │ │ sec/op │ SimpleImage_FetchSquashedContents/docker-archive-4 22.11µ ± 1% SimpleImage_FetchSquashedContents/podman-4 22.05µ ± 0% geomean 22.08µ │ .tmp/benchmark-fb32042.txt │ │ B/op │ SimpleImage_FetchSquashedContents/docker-archive-4 2.648Ki ± 0% SimpleImage_FetchSquashedContents/podman-4 2.648Ki ± 0% geomean 2.648Ki │ .tmp/benchmark-fb32042.txt │ │ allocs/op │ SimpleImage_FetchSquashedContents/docker-archive-4 21.00 ± 0% SimpleImage_FetchSquashedContents/podman-4 21.00 ± 0% geomean 21.00 ```