search

anchore / stereoscope

go library for processing container images and simulating a squash filesystem
Apache License 2.0
78 stars 43 forks source link

Use tar.Header references over copies in tar index #271

Open wagoodman opened 2 months ago

wagoodman commented 2 months ago

This modifies the image metadata and tar index to always use tar.Header references over copies to slightly lower inuse memory utilization. Additionally this removes the sequence field altogether as it is not used.

Inuse memory before the change:

Screenshot 2024-07-11 at 9 13 22 AM

Inuse memory after the change:

Screenshot 2024-07-11 at 9 13 37 AM

Partially addresses #233

github-actions[bot] commented 2 months ago

Benchmark Test Results

Benchmark results from the latest changes vs base branch ``` make .tool/task make[1]: Entering directory '/home/runner/work/stereoscope/stereoscope' make[1]: Leaving directory '/home/runner/work/stereoscope/stereoscope' .tool/task show-benchstat ? github.com/anchore/stereoscope [no test files] ? github.com/anchore/stereoscope/examples [no test files] PASS ok github.com/anchore/stereoscope/internal 0.004s ? github.com/anchore/stereoscope/internal/bus [no test files] PASS ok github.com/anchore/stereoscope/internal/containerd 0.008s PASS ok github.com/anchore/stereoscope/internal/docker 0.005s ? github.com/anchore/stereoscope/internal/log [no test files] PASS ok github.com/anchore/stereoscope/internal/podman 0.005s ? github.com/anchore/stereoscope/pkg/event [no test files] ? github.com/anchore/stereoscope/pkg/event/parsers [no test files] goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor BenchmarkTarIndex-4 33872 35367 ns/op 4128 B/op 93 allocs/op BenchmarkTarIndex-4 33831 35500 ns/op 4130 B/op 93 allocs/op BenchmarkTarIndex-4 33634 35570 ns/op 4129 B/op 93 allocs/op BenchmarkTarIndex-4 33900 35471 ns/op 4129 B/op 93 allocs/op BenchmarkTarIndex-4 34008 36474 ns/op 4129 B/op 93 allocs/op BenchmarkTarIndex-4 33427 35241 ns/op 4128 B/op 93 allocs/op BenchmarkTarIndex-4 33894 35183 ns/op 4127 B/op 93 allocs/op PASS ok github.com/anchore/stereoscope/pkg/file 10.939s PASS ok github.com/anchore/stereoscope/pkg/filetree 0.005s ? github.com/anchore/stereoscope/pkg/filetree/filenode [no test files] PASS ok github.com/anchore/stereoscope/pkg/image 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/containerd 0.009s PASS ok github.com/anchore/stereoscope/pkg/image/docker 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/oci 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/oci/credhelpers 0.005s ? github.com/anchore/stereoscope/pkg/image/podman [no test files] PASS ok github.com/anchore/stereoscope/pkg/image/sif 0.005s ? github.com/anchore/stereoscope/pkg/imagetest [no test files] PASS ok github.com/anchore/stereoscope/pkg/tree 0.004s PASS ok github.com/anchore/stereoscope/pkg/tree/node 0.004s goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/test/integration cpu: AMD EPYC 7763 64-Core Processor BenchmarkSimpleImage_GetImage/docker-archive-4 1096 1066620 ns/op 276076 B/op 2186 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 1123 1062528 ns/op 275669 B/op 2185 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 1129 1063538 ns/op 275562 B/op 2185 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 1118 1055955 ns/op 275466 B/op 2185 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 939 1157617 ns/op 275492 B/op 2185 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 1110 1123801 ns/op 275309 B/op 2185 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 1116 1150876 ns/op 275245 B/op 2185 allocs/op BenchmarkSimpleImage_GetImage/podman-4 62 17668575 ns/op 406247 B/op 2677 allocs/op BenchmarkSimpleImage_GetImage/podman-4 64 17715590 ns/op 405850 B/op 2676 allocs/op BenchmarkSimpleImage_GetImage/podman-4 72 17470192 ns/op 405914 B/op 2675 allocs/op BenchmarkSimpleImage_GetImage/podman-4 69 20634683 ns/op 406019 B/op 2675 allocs/op BenchmarkSimpleImage_GetImage/podman-4 64 17609625 ns/op 406214 B/op 2675 allocs/op BenchmarkSimpleImage_GetImage/podman-4 66 17797945 ns/op 405505 B/op 2674 allocs/op BenchmarkSimpleImage_GetImage/podman-4 63 17461178 ns/op 405163 B/op 2674 allocs/op #0 building with "default" instance using docker driver #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 345B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 transferring context: 2B done #2 DONE 0.0s #3 [internal] load build context #3 transferring context: 209B done #3 DONE 0.0s #4 [2/3] ADD file-2.txt /somefile-2.txt #4 CACHED #5 [1/3] ADD file-1.txt /somefile-1.txt #5 CACHED #6 [3/3] ADD target / #6 CACHED #7 exporting to image #7 exporting layers done #7 writing image sha256:85eaa6e740d88ff4fb7926f34bd6f9c330a97218431acfa27fff8087146478fa done #7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done #7 DONE 0.0s ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" --- FAIL: BenchmarkSimpleImage_GetImage image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2024-07-11 13:21:55.046908121 +0000 UTC, mode: -rw-r--r--) image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7" image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 image_fixtures.go:286: Error Trace: /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33 /home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:163 Error: Received unexpected error: exit status 1 Test: BenchmarkSimpleImage_GetImage Messages: could not import docker image to containerd (shell out) BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53944 22071 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53629 22120 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53712 22127 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54345 22094 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53748 22070 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53962 22046 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54121 22107 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53833 22175 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53628 22187 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54064 22042 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54116 22186 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54070 22153 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53467 22259 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53568 22240 ns/op 2712 B/op 21 allocs/op #0 building with "default" instance using docker driver #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 345B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 transferring context: 2B done #2 DONE 0.0s #3 [internal] load build context #3 transferring context: 209B done #3 DONE 0.0s #4 [1/3] ADD file-1.txt /somefile-1.txt #4 CACHED #5 [2/3] ADD file-2.txt /somefile-2.txt #5 CACHED #6 [3/3] ADD target / #6 CACHED #7 exporting to image #7 exporting layers done #7 writing image sha256:85eaa6e740d88ff4fb7926f34bd6f9c330a97218431acfa27fff8087146478fa done #7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done #7 DONE 0.0s ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" --- FAIL: BenchmarkSimpleImage_FetchSquashedContents image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2024-07-11 13:21:55.046908121 +0000 UTC, mode: -rw-r--r--) image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7" image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 image_fixtures.go:286: Error Trace: /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:64 /home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:189 Error: Received unexpected error: exit status 1 Test: BenchmarkSimpleImage_FetchSquashedContents Messages: could not import docker image to containerd (shell out) FAIL exit status 1 FAIL github.com/anchore/stereoscope/test/integration 38.719s ? github.com/anchore/stereoscope/test/integration/test-fixtures/registry [no test files] FAIL goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor ctr: │ .tmp/benchmark-db573f4.txt │ │ sec/op │ TarIndex-4 35.47µ ± 3% │ .tmp/benchmark-db573f4.txt │ │ B/op │ TarIndex-4 4.032Ki ± 0% │ .tmp/benchmark-db573f4.txt │ │ allocs/op │ TarIndex-4 93.00 ± 0% pkg: github.com/anchore/stereoscope/test/integration │ .tmp/benchmark-db573f4.txt │ │ sec/op │ SimpleImage_GetImage/docker-archive-4 1.067m ± 9% SimpleImage_GetImage/podman-4 17.67m ± 17% geomean 4.341m │ .tmp/benchmark-db573f4.txt │ │ B/op │ SimpleImage_GetImage/docker-archive-4 269.0Ki ± 0% SimpleImage_GetImage/podman-4 396.4Ki ± 0% geomean 326.6Ki │ .tmp/benchmark-db573f4.txt │ │ allocs/op │ SimpleImage_GetImage/docker-archive-4 2.185k ± 0% SimpleImage_GetImage/podman-4 2.675k ± 0% geomean 2.418k ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" │ .tmp/benchmark-db573f4.txt │ │ sec/op │ SimpleImage_FetchSquashedContents/docker-archive-4 22.09µ ± 0% SimpleImage_FetchSquashedContents/podman-4 22.19µ ± 1% geomean 22.14µ │ .tmp/benchmark-db573f4.txt │ │ B/op │ SimpleImage_FetchSquashedContents/docker-archive-4 2.648Ki ± 0% SimpleImage_FetchSquashedContents/podman-4 2.648Ki ± 0% geomean 2.648Ki │ .tmp/benchmark-db573f4.txt │ │ allocs/op │ SimpleImage_FetchSquashedContents/docker-archive-4 21.00 ± 0% SimpleImage_FetchSquashedContents/podman-4 21.00 ± 0% geomean 21.00 goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor ctr: │ .tmp/benchmark-db573f4.txt │ │ sec/op │ TarIndex-4 35.47µ ± 3% │ .tmp/benchmark-db573f4.txt │ │ B/op │ TarIndex-4 4.032Ki ± 0% │ .tmp/benchmark-db573f4.txt │ │ allocs/op │ TarIndex-4 93.00 ± 0% pkg: github.com/anchore/stereoscope/test/integration │ .tmp/benchmark-db573f4.txt │ │ sec/op │ SimpleImage_GetImage/docker-archive-4 1.067m ± 9% SimpleImage_GetImage/podman-4 17.67m ± 17% geomean 4.341m │ .tmp/benchmark-db573f4.txt │ │ B/op │ SimpleImage_GetImage/docker-archive-4 269.0Ki ± 0% SimpleImage_GetImage/podman-4 396.4Ki ± 0% geomean 326.6Ki │ .tmp/benchmark-db573f4.txt │ │ allocs/op │ SimpleImage_GetImage/docker-archive-4 2.185k ± 0% SimpleImage_GetImage/podman-4 2.675k ± 0% geomean 2.418k ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" │ .tmp/benchmark-db573f4.txt │ │ sec/op │ SimpleImage_FetchSquashedContents/docker-archive-4 22.09µ ± 0% SimpleImage_FetchSquashedContents/podman-4 22.19µ ± 1% geomean 22.14µ │ .tmp/benchmark-db573f4.txt │ │ B/op │ SimpleImage_FetchSquashedContents/docker-archive-4 2.648Ki ± 0% SimpleImage_FetchSquashedContents/podman-4 2.648Ki ± 0% geomean 2.648Ki │ .tmp/benchmark-db573f4.txt │ │ allocs/op │ SimpleImage_FetchSquashedContents/docker-archive-4 21.00 ± 0% SimpleImage_FetchSquashedContents/podman-4 21.00 ± 0% geomean 21.00 ```