search

anchore / stereoscope

go library for processing container images and simulating a squash filesystem
Apache License 2.0
78 stars 43 forks source link

Shallow clone trees to save on memory usage #268

Open wagoodman opened 2 months ago

wagoodman commented 2 months ago

Based off of the inuse memory in stereoscope we spend of lot of memory on copying the filetree for squashing. An observation here is that all tree mutation operations always create or replace nodes, but never mutate existing nodes. This means that we can share references across the multiple trees created for the squashfs functionality for each layer.

Here is a snapshot of inuse memory today for image read for a sample image:

Screenshot 2024-07-10 at 1 48 13 PM

Here is a snapshot of inuse memory after using map.Clone to shallow clone existing trees:

Screenshot 2024-07-10 at 1 48 22 PM

Partially fixes #233 in terms of inuse memory

github-actions[bot] commented 2 months ago

Benchmark Test Results

Benchmark results from the latest changes vs base branch ``` make .tool/task make[1]: Entering directory '/home/runner/work/stereoscope/stereoscope' make[1]: Leaving directory '/home/runner/work/stereoscope/stereoscope' .tool/task show-benchstat ? github.com/anchore/stereoscope [no test files] ? github.com/anchore/stereoscope/examples [no test files] PASS ok github.com/anchore/stereoscope/internal 0.004s ? github.com/anchore/stereoscope/internal/bus [no test files] PASS ok github.com/anchore/stereoscope/internal/containerd 0.008s PASS ok github.com/anchore/stereoscope/internal/docker 0.005s ? github.com/anchore/stereoscope/internal/log [no test files] PASS ok github.com/anchore/stereoscope/internal/podman 0.006s ? github.com/anchore/stereoscope/pkg/event [no test files] ? github.com/anchore/stereoscope/pkg/event/parsers [no test files] goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor BenchmarkTarIndex-4 33408 35871 ns/op 5699 B/op 93 allocs/op BenchmarkTarIndex-4 33036 35857 ns/op 5698 B/op 93 allocs/op BenchmarkTarIndex-4 33181 35887 ns/op 5700 B/op 93 allocs/op BenchmarkTarIndex-4 33232 36048 ns/op 5698 B/op 93 allocs/op BenchmarkTarIndex-4 33334 36296 ns/op 5699 B/op 93 allocs/op BenchmarkTarIndex-4 33278 35889 ns/op 5698 B/op 93 allocs/op BenchmarkTarIndex-4 33404 36423 ns/op 5701 B/op 93 allocs/op PASS ok github.com/anchore/stereoscope/pkg/file 10.963s PASS ok github.com/anchore/stereoscope/pkg/filetree 0.005s ? github.com/anchore/stereoscope/pkg/filetree/filenode [no test files] PASS ok github.com/anchore/stereoscope/pkg/image 0.005s PASS ok github.com/anchore/stereoscope/pkg/image/containerd 0.009s PASS ok github.com/anchore/stereoscope/pkg/image/docker 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/oci 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/oci/credhelpers 0.006s ? github.com/anchore/stereoscope/pkg/image/podman [no test files] PASS ok github.com/anchore/stereoscope/pkg/image/sif 0.005s ? github.com/anchore/stereoscope/pkg/imagetest [no test files] PASS ok github.com/anchore/stereoscope/pkg/tree 0.004s PASS ok github.com/anchore/stereoscope/pkg/tree/node 0.003s goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/test/integration cpu: AMD EPYC 7763 64-Core Processor BenchmarkSimpleImage_GetImage/docker-archive-4 986 1198326 ns/op 286923 B/op 2405 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 1009 1187552 ns/op 286492 B/op 2404 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 1003 1193542 ns/op 286417 B/op 2404 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 988 1187906 ns/op 286379 B/op 2404 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 1000 1311893 ns/op 286282 B/op 2404 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 996 1248665 ns/op 286191 B/op 2404 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 1012 1189311 ns/op 286216 B/op 2404 allocs/op BenchmarkSimpleImage_GetImage/podman-4 66 18385907 ns/op 408690 B/op 2661 allocs/op BenchmarkSimpleImage_GetImage/podman-4 67 18230953 ns/op 408889 B/op 2660 allocs/op BenchmarkSimpleImage_GetImage/podman-4 62 18978946 ns/op 409169 B/op 2662 allocs/op BenchmarkSimpleImage_GetImage/podman-4 63 18732079 ns/op 408366 B/op 2661 allocs/op BenchmarkSimpleImage_GetImage/podman-4 66 18087372 ns/op 407812 B/op 2660 allocs/op BenchmarkSimpleImage_GetImage/podman-4 67 18316621 ns/op 408939 B/op 2662 allocs/op BenchmarkSimpleImage_GetImage/podman-4 68 19174047 ns/op 408710 B/op 2660 allocs/op #0 building with "default" instance using docker driver #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 345B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 transferring context: 2B done #2 DONE 0.0s #3 [internal] load build context #3 transferring context: 209B done #3 DONE 0.0s #4 [1/3] ADD file-1.txt /somefile-1.txt #4 CACHED #5 [2/3] ADD file-2.txt /somefile-2.txt #5 CACHED #6 [3/3] ADD target / #6 CACHED #7 exporting to image #7 exporting layers done #7 writing image sha256:112a14620f24a6696f8709eaed159438b8298ad846515ed17ff857dd7fb4bff1 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done #7 DONE 0.0s ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" --- FAIL: BenchmarkSimpleImage_GetImage image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2024-07-15 13:59:39.053549977 +0000 UTC, mode: -rw-r--r--) image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7" image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 image_fixtures.go:286: Error Trace: /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33 /home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:163 Error: Received unexpected error: exit status 1 Test: BenchmarkSimpleImage_GetImage Messages: could not import docker image to containerd (shell out) BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53662 22234 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54211 22211 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53786 22222 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53091 22204 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53756 22217 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53806 22230 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 53996 22280 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53552 22218 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53845 22218 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53950 22222 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 50258 22500 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53922 22175 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53571 22214 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53755 22725 ns/op 2712 B/op 21 allocs/op #0 building with "default" instance using docker driver #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 345B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 transferring context: 2B done #2 DONE 0.0s #3 [internal] load build context #3 transferring context: 209B done #3 DONE 0.0s #4 [1/3] ADD file-1.txt /somefile-1.txt #4 CACHED #5 [2/3] ADD file-2.txt /somefile-2.txt #5 CACHED #6 [3/3] ADD target / #6 CACHED #7 exporting to image #7 exporting layers done #7 writing image sha256:112a14620f24a6696f8709eaed159438b8298ad846515ed17ff857dd7fb4bff1 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done #7 DONE 0.0s ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" --- FAIL: BenchmarkSimpleImage_FetchSquashedContents image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2024-07-15 13:59:39.053549977 +0000 UTC, mode: -rw-r--r--) image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7" image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 image_fixtures.go:286: Error Trace: /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:64 /home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:189 Error: Received unexpected error: exit status 1 Test: BenchmarkSimpleImage_FetchSquashedContents Messages: could not import docker image to containerd (shell out) FAIL exit status 1 FAIL github.com/anchore/stereoscope/test/integration 39.666s ? github.com/anchore/stereoscope/test/integration/test-fixtures/registry [no test files] FAIL goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor ctr: │ .tmp/benchmark-738d12d.txt │ │ sec/op │ TarIndex-4 35.89µ ± 1% │ .tmp/benchmark-738d12d.txt │ │ B/op │ TarIndex-4 5.565Ki ± 0% │ .tmp/benchmark-738d12d.txt │ │ allocs/op │ TarIndex-4 93.00 ± 0% pkg: github.com/anchore/stereoscope/test/integration │ .tmp/benchmark-738d12d.txt │ │ sec/op │ SimpleImage_GetImage/docker-archive-4 1.194m ± 10% SimpleImage_GetImage/podman-4 18.39m ± 4% geomean 4.684m │ .tmp/benchmark-738d12d.txt │ │ B/op │ SimpleImage_GetImage/docker-archive-4 279.7Ki ± 0% SimpleImage_GetImage/podman-4 399.1Ki ± 0% geomean 334.1Ki │ .tmp/benchmark-738d12d.txt │ │ allocs/op │ SimpleImage_GetImage/docker-archive-4 2.404k ± 0% SimpleImage_GetImage/podman-4 2.661k ± 0% geomean 2.529k ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" │ .tmp/benchmark-738d12d.txt │ │ sec/op │ SimpleImage_FetchSquashedContents/docker-archive-4 22.22µ ± 0% SimpleImage_FetchSquashedContents/podman-4 22.22µ ± 2% geomean 22.22µ │ .tmp/benchmark-738d12d.txt │ │ B/op │ SimpleImage_FetchSquashedContents/docker-archive-4 2.648Ki ± 0% SimpleImage_FetchSquashedContents/podman-4 2.648Ki ± 0% geomean 2.648Ki │ .tmp/benchmark-738d12d.txt │ │ allocs/op │ SimpleImage_FetchSquashedContents/docker-archive-4 21.00 ± 0% SimpleImage_FetchSquashedContents/podman-4 21.00 ± 0% geomean 21.00 goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor ctr: │ .tmp/benchmark-738d12d.txt │ │ sec/op │ TarIndex-4 35.89µ ± 1% │ .tmp/benchmark-738d12d.txt │ │ B/op │ TarIndex-4 5.565Ki ± 0% │ .tmp/benchmark-738d12d.txt │ │ allocs/op │ TarIndex-4 93.00 ± 0% pkg: github.com/anchore/stereoscope/test/integration │ .tmp/benchmark-738d12d.txt │ │ sec/op │ SimpleImage_GetImage/docker-archive-4 1.194m ± 10% SimpleImage_GetImage/podman-4 18.39m ± 4% geomean 4.684m │ .tmp/benchmark-738d12d.txt │ │ B/op │ SimpleImage_GetImage/docker-archive-4 279.7Ki ± 0% SimpleImage_GetImage/podman-4 399.1Ki ± 0% geomean 334.1Ki │ .tmp/benchmark-738d12d.txt │ │ allocs/op │ SimpleImage_GetImage/docker-archive-4 2.404k ± 0% SimpleImage_GetImage/podman-4 2.661k ± 0% geomean 2.529k ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" │ .tmp/benchmark-738d12d.txt │ │ sec/op │ SimpleImage_FetchSquashedContents/docker-archive-4 22.22µ ± 0% SimpleImage_FetchSquashedContents/podman-4 22.22µ ± 2% geomean 22.22µ │ .tmp/benchmark-738d12d.txt │ │ B/op │ SimpleImage_FetchSquashedContents/docker-archive-4 2.648Ki ± 0% SimpleImage_FetchSquashedContents/podman-4 2.648Ki ± 0% geomean 2.648Ki │ .tmp/benchmark-738d12d.txt │ │ allocs/op │ SimpleImage_FetchSquashedContents/docker-archive-4 21.00 ± 0% SimpleImage_FetchSquashedContents/podman-4 21.00 ± 0% geomean 21.00 ```