search

anchore / stereoscope

go library for processing container images and simulating a squash filesystem
Apache License 2.0
81 stars 44 forks source link

Fix panic when pulling OCI-packaged helm chart #228

Closed willmurphyscode closed 6 months ago

willmurphyscode commented 7 months ago

Reported in https://github.com/anchore/syft/issues/2745

TODO

❯ go run cmd/syft/main.go bitnamicharts/nginx:15.14.1
 ✔ Pulled image                    
 ✔ Parsed image                                                                                                                   sha256:fa265257e1b905d79242f2b25b506057d179a108831b99f27a2085fc02706fff
could not determine source: errors occurred attempting to resolve 'bitnamicharts/nginx:15.14.1':
  - no such file or directory
  - unable to inspect existing image: Error response from daemon: No such image: bitnamicharts/nginx:15.14.1
  - podman not available: making http client: connection to bastion host="127.0.0.1:57447" failed: dial tcp 127.0.0.1:57447: connect: connection refused
  - containerd not available: no grpc connection or services is available: unavailable
  - unknown layer media type: application/vnd.cncf.helm.chart.content.v1.tar+gzip
exit status 1

IMO this is sort of confusing output. The last line "unknown layer media type: application/vnd.cncf.helm.chart.content.v1.tar+gzip" seems the most informative.

Basically, this is the error because stereoscope tried to use all its image providers to pull the image, and they all failed with various things, and Syft prints them all because it doesn't know which one is the "real" error. Probably more work is needed to translate this to a higher level error message in Syft. For example, the error has nothing to do with podman not being installed, but that's reported with the same UI prominence as the unsupported media type error.

github-actions[bot] commented 7 months ago

Benchmark Test Results

Benchmark results from the latest changes vs base branch ``` make .tool/task make[1]: Entering directory '/home/runner/work/stereoscope/stereoscope' make[1]: Leaving directory '/home/runner/work/stereoscope/stereoscope' .tool/task show-benchstat ? github.com/anchore/stereoscope [no test files] ? github.com/anchore/stereoscope/examples [no test files] PASS ok github.com/anchore/stereoscope/internal 0.004s ? github.com/anchore/stereoscope/internal/bus [no test files] PASS ok github.com/anchore/stereoscope/internal/containerd 0.007s PASS ok github.com/anchore/stereoscope/internal/docker 0.005s ? github.com/anchore/stereoscope/internal/log [no test files] PASS ok github.com/anchore/stereoscope/internal/podman 0.005s ? github.com/anchore/stereoscope/pkg/event [no test files] ? github.com/anchore/stereoscope/pkg/event/parsers [no test files] goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor BenchmarkTarIndex-4 33177 36191 ns/op 5699 B/op 93 allocs/op BenchmarkTarIndex-4 31023 36061 ns/op 5700 B/op 93 allocs/op BenchmarkTarIndex-4 33148 36171 ns/op 5701 B/op 93 allocs/op BenchmarkTarIndex-4 32860 36280 ns/op 5701 B/op 93 allocs/op BenchmarkTarIndex-4 32990 36122 ns/op 5701 B/op 93 allocs/op BenchmarkTarIndex-4 32941 36155 ns/op 5701 B/op 93 allocs/op BenchmarkTarIndex-4 32884 36144 ns/op 5699 B/op 93 allocs/op PASS ok github.com/anchore/stereoscope/pkg/file 10.897s PASS ok github.com/anchore/stereoscope/pkg/filetree 0.005s ? github.com/anchore/stereoscope/pkg/filetree/filenode [no test files] PASS ok github.com/anchore/stereoscope/pkg/image 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/containerd 0.010s PASS ok github.com/anchore/stereoscope/pkg/image/docker 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/oci 0.006s PASS ok github.com/anchore/stereoscope/pkg/image/oci/credhelpers 0.005s ? github.com/anchore/stereoscope/pkg/image/podman [no test files] PASS ok github.com/anchore/stereoscope/pkg/image/sif 0.005s ? github.com/anchore/stereoscope/pkg/imagetest [no test files] PASS ok github.com/anchore/stereoscope/pkg/tree 0.003s PASS ok github.com/anchore/stereoscope/pkg/tree/node 0.003s goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/test/integration cpu: AMD EPYC 7763 64-Core Processor BenchmarkSimpleImage_GetImage/docker-archive-4 909 1405043 ns/op 344294 B/op 2870 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 823 1286558 ns/op 343827 B/op 2869 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 928 1281694 ns/op 343798 B/op 2869 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 928 1289905 ns/op 343745 B/op 2869 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 930 1309042 ns/op 343582 B/op 2869 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 933 1286697 ns/op 343699 B/op 2869 allocs/op BenchmarkSimpleImage_GetImage/docker-archive-4 930 1336920 ns/op 343486 B/op 2869 allocs/op BenchmarkSimpleImage_GetImage/podman-4 68 17540879 ns/op 456320 B/op 2809 allocs/op BenchmarkSimpleImage_GetImage/podman-4 67 17483415 ns/op 454597 B/op 2809 allocs/op BenchmarkSimpleImage_GetImage/podman-4 68 17509107 ns/op 453750 B/op 2808 allocs/op BenchmarkSimpleImage_GetImage/podman-4 68 18630610 ns/op 453769 B/op 2808 allocs/op BenchmarkSimpleImage_GetImage/podman-4 69 17669483 ns/op 454033 B/op 2809 allocs/op BenchmarkSimpleImage_GetImage/podman-4 62 17425419 ns/op 454029 B/op 2808 allocs/op BenchmarkSimpleImage_GetImage/podman-4 68 17535314 ns/op 454296 B/op 2808 allocs/op #0 building with "default" instance using docker driver #1 [internal] load .dockerignore #1 transferring context: 2B done #1 DONE 0.0s #2 [internal] load build definition from Dockerfile #2 transferring dockerfile: 345B done #2 DONE 0.0s #3 [internal] load build context #3 transferring context: 209B done #3 DONE 0.0s #4 [2/3] ADD file-2.txt /somefile-2.txt #4 CACHED #5 [1/3] ADD file-1.txt /somefile-1.txt #5 CACHED #6 [3/3] ADD target / #6 CACHED #7 exporting to image #7 exporting layers done #7 writing image sha256:9305056fdb21c64daf0634548364bfa4348ae9e0817ba07c0a973f4b097cf858 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done #7 DONE 0.0s ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" --- FAIL: BenchmarkSimpleImage_GetImage image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 21504, modified: 2024-04-23 16:20:39.04005625 +0000 UTC, mode: -rw-r--r--) image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7" image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 image_fixtures.go:286: Error Trace: /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33 /home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:163 Error: Received unexpected error: exit status 1 Test: BenchmarkSimpleImage_GetImage Messages: could not import docker image to containerd (shell out) BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54661 21759 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54654 21887 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54584 21913 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54784 22059 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54292 22006 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54327 22085 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4 54510 22101 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53926 21992 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54654 22528 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54742 22029 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54207 22020 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 54224 22059 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53247 22008 ns/op 2712 B/op 21 allocs/op BenchmarkSimpleImage_FetchSquashedContents/podman-4 53805 22021 ns/op 2712 B/op 21 allocs/op #0 building with "default" instance using docker driver #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 345B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 transferring context: 2B done #2 DONE 0.0s #3 [internal] load build context #3 transferring context: 209B done #3 DONE 0.0s #4 [1/3] ADD file-1.txt /somefile-1.txt #4 CACHED #5 [2/3] ADD file-2.txt /somefile-2.txt #5 CACHED #6 [3/3] ADD target / #6 CACHED #7 exporting to image #7 exporting layers done #7 writing image sha256:9305056fdb21c64daf0634548364bfa4348ae9e0817ba07c0a973f4b097cf858 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done #7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done #7 DONE 0.0s ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" --- FAIL: BenchmarkSimpleImage_FetchSquashedContents image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 21504, modified: 2024-04-23 16:20:39.04005625 +0000 UTC, mode: -rw-r--r--) image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7" image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 image_fixtures.go:286: Error Trace: /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33 /home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:64 /home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:189 Error: Received unexpected error: exit status 1 Test: BenchmarkSimpleImage_FetchSquashedContents Messages: could not import docker image to containerd (shell out) FAIL exit status 1 FAIL github.com/anchore/stereoscope/test/integration 39.277s ? github.com/anchore/stereoscope/test/integration/test-fixtures/registry [no test files] FAIL goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor ctr: │ .tmp/benchmark-7879247.txt │ │ sec/op │ TarIndex-4 36.16µ ± 0% │ .tmp/benchmark-7879247.txt │ │ B/op │ TarIndex-4 5.567Ki ± 0% │ .tmp/benchmark-7879247.txt │ │ allocs/op │ TarIndex-4 93.00 ± 0% pkg: github.com/anchore/stereoscope/test/integration │ .tmp/benchmark-7879247.txt │ │ sec/op │ SimpleImage_GetImage/docker-archive-4 1.290m ± 9% SimpleImage_GetImage/podman-4 17.54m ± 6% geomean 4.756m │ .tmp/benchmark-7879247.txt │ │ B/op │ SimpleImage_GetImage/docker-archive-4 335.7Ki ± 0% SimpleImage_GetImage/podman-4 443.4Ki ± 1% geomean 385.8Ki │ .tmp/benchmark-7879247.txt │ │ allocs/op │ SimpleImage_GetImage/docker-archive-4 2.869k ± 0% SimpleImage_GetImage/podman-4 2.808k ± 0% geomean 2.838k ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" │ .tmp/benchmark-7879247.txt │ │ sec/op │ SimpleImage_FetchSquashedContents/docker-archive-4 22.01µ ± 1% SimpleImage_FetchSquashedContents/podman-4 22.02µ ± 2% geomean 22.01µ │ .tmp/benchmark-7879247.txt │ │ B/op │ SimpleImage_FetchSquashedContents/docker-archive-4 2.648Ki ± 0% SimpleImage_FetchSquashedContents/podman-4 2.648Ki ± 0% geomean 2.648Ki │ .tmp/benchmark-7879247.txt │ │ allocs/op │ SimpleImage_FetchSquashedContents/docker-archive-4 21.00 ± 0% SimpleImage_FetchSquashedContents/podman-4 21.00 ± 0% geomean 21.00 goos: linux goarch: amd64 pkg: github.com/anchore/stereoscope/pkg/file cpu: AMD EPYC 7763 64-Core Processor ctr: │ .tmp/benchmark-7879247.txt │ │ sec/op │ TarIndex-4 36.16µ ± 0% │ .tmp/benchmark-7879247.txt │ │ B/op │ TarIndex-4 5.567Ki ± 0% │ .tmp/benchmark-7879247.txt │ │ allocs/op │ TarIndex-4 93.00 ± 0% pkg: github.com/anchore/stereoscope/test/integration │ .tmp/benchmark-7879247.txt │ │ sec/op │ SimpleImage_GetImage/docker-archive-4 1.290m ± 9% SimpleImage_GetImage/podman-4 17.54m ± 6% geomean 4.756m │ .tmp/benchmark-7879247.txt │ │ B/op │ SimpleImage_GetImage/docker-archive-4 335.7Ki ± 0% SimpleImage_GetImage/podman-4 443.4Ki ± 1% geomean 385.8Ki │ .tmp/benchmark-7879247.txt │ │ allocs/op │ SimpleImage_GetImage/docker-archive-4 2.869k ± 0% SimpleImage_GetImage/podman-4 2.808k ± 0% geomean 2.838k ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied" │ .tmp/benchmark-7879247.txt │ │ sec/op │ SimpleImage_FetchSquashedContents/docker-archive-4 22.01µ ± 1% SimpleImage_FetchSquashedContents/podman-4 22.02µ ± 2% geomean 22.01µ │ .tmp/benchmark-7879247.txt │ │ B/op │ SimpleImage_FetchSquashedContents/docker-archive-4 2.648Ki ± 0% SimpleImage_FetchSquashedContents/podman-4 2.648Ki ± 0% geomean 2.648Ki │ .tmp/benchmark-7879247.txt │ │ allocs/op │ SimpleImage_FetchSquashedContents/docker-archive-4 21.00 ± 0% SimpleImage_FetchSquashedContents/podman-4 21.00 ± 0% geomean 21.00 ```
willmurphyscode commented 7 months ago

The new syft error handling (PR soon) will look like this:

❯ go run cmd/syft/main.go bitnamicharts/nginx:15.14.1
 ✔ Pulled image                    
 ✔ Parsed image                                                                                                                                                                                                                                                                                                                                                                                           sha256:fa265257e1b905d79242f2b25b506057d179a108831b99f27a2085fc02706fff
could not determine source: errors occurred attempting to resolve 'bitnamicharts/nginx:15.14.1':
  - docker: unable to inspect existing image: Error response from daemon: No such image: bitnamicharts/nginx:15.14.1
  - podman: podman not available: making http client: connection to bastion host="127.0.0.1:57447" failed: dial tcp 127.0.0.1:57447: connect: connection refused
  - containerd: containerd not available: no grpc connection or services is available: unavailable
  - oci-registry: unknown layer media type: application/vnd.cncf.helm.chart.content.v1.tar+gzip
  - additionally, the following providers failed with file does not exist: docker-archive, oci-archive, oci-dir, singularity, oci-dir, local-file, local-directory

Because we don't know which provider the user was expecting to handle the source, report each error with the name of the provider that caused it, except for providers that fail with a file not found error - just report a list of those, since every provider that assumes the input string was a path on the local filesystem will fail with that error if it wasn't.