Open markszabo opened 2 years ago
Can you share the invalid tar file, and the valid one produced by skopeo copying? Diffing them may help uncover what the problem is.
Sure! Here is the file produced by kaniko: out.tar.gz and here is after copying with skopeo: out2.tar.gz (both got renamed to .tar.gz
from .tar
as Github doesn't support .tar
files)
I think I had a meeting with developers form Snyk . We spoke about changing the kaniko tar format to adhere to https://snyk.io/blog/container-image-formats.
However, we havent seen any movement from them.
I think this issue is also related to scanning docker images using Synopsis Black Duck, I am getting a similar issue " Exception thrown while getting image packages: Could not parse the image manifest file"
There was a potentially similar issue with Dive (https://github.com/wagoodman/dive/issues/318).
Docker and Kaniko use different naming conventions for the config files. Dive assumed all config files would end in .json
and would filter based on name. Kaniko uses the format sha254:<hash>
as the file name and was filtered. Additionally, Kaniko stores layers as tar.gz
, rather than plain tar
files.
Looking at @markszabo's example files, we can see similar differences:
> tar xvf out.tar.gz -C kaniko/
sha256:6f4eeeb6af6975a3a005943d019f2149d3d90815b6403ab11d527f8b21baf26a
59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3.tar.gz
39c46de6fa0d18af0c4239463ac9525d38f3d4bc1bd373a703f3699c31aa68df.tar.gz
manifest.json
> tar xvf out2.tar.gz -C skopeo/
8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759.tar
5d8df4285410a06dc1639e7e808e90b0a25508ed5d01ad9d02e868c95299f924.tar
6f4eeeb6af6975a3a005943d019f2149d3d90815b6403ab11d527f8b21baf26a.json
fe012f240489f197d98045542108493bb5fc3b951ddfa835c7f1c2867edde3d6/layer.tar
fe012f240489f197d98045542108493bb5fc3b951ddfa835c7f1c2867edde3d6/VERSION
fe012f240489f197d98045542108493bb5fc3b951ddfa835c7f1c2867edde3d6/json
1425cd74f86c80c63fcc46f145453581a1256700db0aad28cfb55d139e1c5066/layer.tar
1425cd74f86c80c63fcc46f145453581a1256700db0aad28cfb55d139e1c5066/VERSION
1425cd74f86c80c63fcc46f145453581a1256700db0aad28cfb55d139e1c5066/json
manifest.json
repositories
Is there any further update on this issue?
Any news on this??
snyk appears to fail on ~50% of images. Another: docker.io/library/busybox:latest
via Docker Desktop UI on Apple silicon.
does Synk supports or scans kaniko produced images ? any further update on this ?
I'm seeing the same issue on my end, any updates?
I would like to see this fixed, one way or another, as well please.
The only way I was able to get this to work is by doing the below, the solution assumed you are using aws ecr as the repo & kaniko image to build out your docker images.
Automation Pipelines General Overview:
This is an obvious work around, but since the team has no direction as to when it will get fixed, this is the solution I'm leveraging for automated docker scans in the pipeline.
The entire solution can be put into a docker container btw, i.e install the snyk cli + skopeo tool on a docker container.
Command for skopeo to get the ECR Token and make the call to aws ecr to pull the image layers: def ECR_TOKEN = sh(script: "aws ecr get-login-password --region ${DOCKER_REGION} --profile ${AWS_PROFILE}", returnStdout: true).trim() sh "set +x && skopeo copy --src-creds=AWS:${ECR_TOKEN} docker://${DOCKER_REGISTRY}/${DOCKER_REPOSITORY}:${IMAGE_TAG} oci-archive:image.tar"
@cmanfre4 @svilambi @skull-squadron @R3DRUN3 @MatthiasRoels
I was able to get it to work by doing skopeo copy docker-archive:$tar_from_kaniko oci-archive:$oci_archive_out
which doesn't need ECR or any remote registry.
If anybody is looking for an example repro, have a kaniko built image which snyk container test
fails to scan. The skopeo copy
workaround does work, but I have to use docker-archive:...
for both the input and output (if I try oci-archive:
for the output snyk still cannot scan the resulting tar file).
Source: https://github.com/CMS-Enterprise/batcave-knight-light/tree/main/node-server
Build with gcr.io/kaniko-project/executor:v1.23.2
on an arm64 arch using the command --context=/app --dockerfile=/app/Dockerfile --verbosity=info --no-push --tar-path=/out/image.tar
Attempting to scan with a locally build version of the snky/snky:alpine
image derived from v1.1291.1
(see https://github.com/sflanker/snyk-cli/tree/feat/alpine-arm64-support), using the CLI: snyk container test --exclude-app-vulns --severity-threshold=high docker-archive:image.tar
).
Initially I received Invalid OCI archive
.
After running skopeo copy docker-archive:image.tar docker-archive:image.docker-to-docker.tar
the scan succeeded.
Original archive: https://drive.google.com/file/d/12k0ovaUfDfhnzqkPzksTIqi3gs_7eA4R/view?usp=drive_link Converted archive: https://drive.google.com/file/d/1AeAT01ZZwX8UxEc0A_dA9siv_G0D2bjm/view?usp=sharing
The issue is also reproducible in a CI environment running on amd64 instead of arm.
It seems likely that this is a problem with snyk, not with Kaniko. Snyk has the same issue with images built using buildx with buildkit and save/exported using the --output type=docker,dest=./image.tar
argument.
Actual behavior
Snyk fails to scan docker images produced by kaniko:
However
docker load
ing the same image make snyk work:Expected behavior
I expect kaniko to produce images that can be scanned with snyk
To Reproduce
See above.
Additional Information
Workaround
We found that if the archive is copied with
skopeo
, then snyk can work with it:Triage Notes for the Maintainers
--cache
flag