search

rancher / wharfie

Pulls and unpacks a container image to the local filesystem. Usable both as a standalone CLI tool, and as a go module.
Apache License 2.0
25 stars 14 forks source link

Fix path handling and add e2e image extract test #8

Closed brandond closed 2 years ago

brandond commented 2 years ago

Fixes a regression caused by updating go-containerregistry, which causes tar entry paths to be normalized to use os-specific path separators: https://github.com/google/go-containerregistry/commit/f9a1886f3

Signed-off-by: Brad Davidson brad.davidson@rancher.com

Tested locally on Windows, both via go test and pulled through into RKE2.

Windows test output:

?       github.com/rancher/wharfie/pkg/credentialprovider/plugin    [no test files]
=== RUN   TestFindPathFromExtract
    extract_test.go:45: Testing ExtractDirs with map map[string]string{"/bin":"C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin", "/charts":"C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts"} for linux
time="2021-12-21T23:46:49-08:00" level=info msg="Creating directory C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin"
time="2021-12-21T23:46:49-08:00" level=info msg="Extracting file bin\\containerd to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\containerd"
time="2021-12-21T23:46:49-08:00" level=info msg="Extracting file bin\\containerd-shim to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\containerd-shim"
time="2021-12-21T23:46:50-08:00" level=info msg="Extracting file bin\\containerd-shim-runc-v1 to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\containerd-shim-runc-v1"
time="2021-12-21T23:46:50-08:00" level=info msg="Extracting file bin\\containerd-shim-runc-v2 to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\containerd-shim-runc-v2"
time="2021-12-21T23:46:50-08:00" level=info msg="Extracting file bin\\crictl to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\crictl"
time="2021-12-21T23:46:51-08:00" level=info msg="Extracting file bin\\ctr to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\ctr"
time="2021-12-21T23:46:51-08:00" level=info msg="Extracting file bin\\kubectl to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\kubectl"
time="2021-12-21T23:46:52-08:00" level=info msg="Extracting file bin\\kubelet to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\kubelet"
time="2021-12-21T23:46:53-08:00" level=info msg="Extracting file bin\\runc to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\runc"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file bin\\socat to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\socat"
time="2021-12-21T23:46:54-08:00" level=info msg="Creating directory C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\harvester-cloud-provider.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\harvester-cloud-provider.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\harvester-csi-driver.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\harvester-csi-driver.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rancher-vsphere-cpi.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rancher-vsphere-cpi.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rancher-vsphere-csi.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rancher-vsphere-csi.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rke2-calico-crd.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rke2-calico-crd.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rke2-calico.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rke2-calico.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rke2-canal.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rke2-canal.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rke2-cilium.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rke2-cilium.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rke2-coredns.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rke2-coredns.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rke2-ingress-nginx.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rke2-ingress-nginx.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rke2-metrics-server.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rke2-metrics-server.yaml"
time="2021-12-21T23:46:54-08:00" level=info msg="Extracting file charts\\rke2-multus.yaml to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts\\rke2-multus.yaml"
    extract_test.go:57: containerd binary for linux extracted successfully: containerd
    extract_test.go:45: Testing ExtractDirs with map map[string]string{"/bin":"C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin", "/charts":"C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\charts"} for windows
time="2021-12-21T23:46:55-08:00" level=info msg="Creating directory C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin"
time="2021-12-21T23:46:55-08:00" level=info msg="Extracting file bin\\calico-ipam.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\calico-ipam.exe"
time="2021-12-21T23:46:56-08:00" level=info msg="Extracting file bin\\calico-node.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\calico-node.exe"
time="2021-12-21T23:46:57-08:00" level=info msg="Extracting file bin\\calico.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\calico.exe"
time="2021-12-21T23:46:57-08:00" level=info msg="Extracting file bin\\containerd-shim-runhcs-v1.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\containerd-shim-runhcs-v1.exe"
time="2021-12-21T23:46:58-08:00" level=info msg="Extracting file bin\\containerd.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\containerd.exe"
time="2021-12-21T23:46:58-08:00" level=info msg="Extracting file bin\\crictl.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\crictl.exe"
time="2021-12-21T23:46:59-08:00" level=info msg="Extracting file bin\\ctr.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\ctr.exe"
time="2021-12-21T23:46:59-08:00" level=info msg="Extracting file bin\\hns.psm1 to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\hns.psm1"
time="2021-12-21T23:46:59-08:00" level=info msg="Extracting file bin\\host-local.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\host-local.exe"
time="2021-12-21T23:46:59-08:00" level=info msg="Extracting file bin\\kube-proxy.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\kube-proxy.exe"
time="2021-12-21T23:47:00-08:00" level=info msg="Extracting file bin\\kubectl.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\kubectl.exe"
time="2021-12-21T23:47:01-08:00" level=info msg="Extracting file bin\\kubelet.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\kubelet.exe"
time="2021-12-21T23:47:02-08:00" level=info msg="Extracting file bin\\win-overlay.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\win-overlay.exe"
time="2021-12-21T23:47:03-08:00" level=info msg="Extracting file bin\\wins.exe to C:\\Users\\brandond\\AppData\\Local\\Temp\\TestFindPathFromExtract2648320399\\001\\bin\\wins.exe"
    extract_test.go:57: containerd binary for windows extracted successfully: containerd.exe
--- PASS: TestFindPathFromExtract (15.42s)
=== RUN   TestFindPath
    extract_test.go:187: Testing paths with dirs extract.mss{"/":"C:\\Users\\brandond\\AppData\\Local\\Temp"}
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\test.txt", err <nil> for image path "\\test.txt"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\test.txt", err <nil> for image path "\\test.txt"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\test.txt", err <nil> for image path "\\test.txt"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\test.txt", err <nil> for image path "test.txt"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\etc\\hosts", err <nil> for image path "\\etc\\hosts"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\var\\lib\\rancher", err <nil> for image path "\\var\\lib\\rancher"
    extract_test.go:197: Got mapped path "", err illegal path for image path "..\\..\\etc\\passwd"
    extract_test.go:187: Testing paths with dirs extract.mss{}
    extract_test.go:197: Got mapped path "", err <nil> for image path "\\text.txt"
    extract_test.go:187: Testing paths with dirs extract.mss{"/Files":"C:\\Users\\brandond\\AppData\\Local\\Temp\\Files", "/Files/bin":"C:\\Users\\brandond\\AppData\\Local\\Temp\\Files-bin", "/etc":"C:\\Users\\brandond\\AppData\\Local\\Temp\\etc"}
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\Files-bin", err <nil> for image path "Files\\bin"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\Files-bin\\test.txt", err <nil> for image path "Files\\bin\\test.txt"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\Files-bin\\aux", err <nil> for image path "Files\\bin\\aux"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\Files-bin\\aux\\mount", err <nil> for image path "Files\\bin\\aux\\mount"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\Files", err <nil> for image path "Files"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\Files\\test.txt", err <nil> for image path "Files\\test.txt"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\Files\\opt", err <nil> for image path "Files\\opt"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\Files\\opt\\other.txt", err <nil> for image path "Files\\opt\\other.txt"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\etc", err <nil> for image path "etc"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\etc\\hosts", err <nil> for image path "etc\\hosts"
    extract_test.go:197: Got mapped path "C:\\Users\\brandond\\AppData\\Local\\Temp\\etc\\shadow\\passwd", err <nil> for image path "etc\\shadow\\passwd"
    extract_test.go:197: Got mapped path "", err <nil> for image path "sbin"
    extract_test.go:197: Got mapped path "", err <nil> for image path "sbin\\ip"
    extract_test.go:197: Got mapped path "", err illegal path for image path "..\\..\\etc\\passwd"
--- PASS: TestFindPath (0.00s)
PASS
ok      github.com/rancher/wharfie/pkg/extract  (cached)
=== RUN   TestRewrite
time="2021-12-21T23:46:48-08:00" level=warning msg="Failed to compile rewrite `(.*` for index.docker.io"
    registries_test.go:156: OK  rewrite for busybox as index.docker.io/library/busybox:latest - got index.docker.io/library/busybox:latest
    registries_test.go:156: OK  rewrite for busybox as index.docker.io/library/busybox:latest - got index.docker.io/library/busybox:latest
    registries_test.go:156: OK  rewrite for registry.local/test as registry.local/test:latest - got registry.local/test:latest
    registries_test.go:156: OK  rewrite for busybox as index.docker.io/library/busybox:latest - got index.docker.io/docker/library/busybox:latest
    registries_test.go:156: OK  rewrite for registry.local/test as registry.local/test:latest - got registry.local/test:latest
    registries_test.go:156: OK  rewrite for busybox@sha256:82becede498899ec668628e7cb0ad87b6e1c371cb8a1e597d83a47fac21d6af3 as index.docker.io/library/busybox@sha256:82becede498899ec668628e7cb0ad87b6e1c371cb8a1e597d83a47fac21d6af3 - got index.docker.io/docker/library/busybox@sha256:82becede498899ec668628e7cb0ad87b6e1c371cb8a1e597d83a47fac21d6af3
    registries_test.go:156: OK  rewrite for busybox as index.docker.io/library/busybox:latest - got index.docker.io/library/busybox:latest
    registries_test.go:156: OK  rewrite for registry.local/test as registry.local/test:latest - got registry.local/localimages/test:latest
    registries_test.go:156: OK  rewrite for rancher/rancher:v2.5.9 as index.docker.io/rancher/rancher:v2.5.9 - got index.docker.io/rancher/prod/rancher:v2.5.9
    registries_test.go:156: OK  rewrite for longhornio/longhorn-engine:v1.1.1 as index.docker.io/longhornio/longhorn-engine:v1.1.1 - got index.docker.io/longhornio/staging/longhorn-engine:v1.1.1
    registries_test.go:156: OK  rewrite for busybox as index.docker.io/library/busybox:latest - got index.docker.io/library/busybox:latest
    registries_test.go:156: OK  rewrite for busybox as index.docker.io/library/busybox:latest - got index.docker.io/docker.io/library/busybox:latest
    registries_test.go:156: OK  rewrite for registry.local/test as registry.local/test:latest - got registry.local/test:latest
    registries_test.go:156: OK  rewrite for busybox as index.docker.io/library/busybox:latest - got index.docker.io/library/busybox:latest
    registries_test.go:156: OK  rewrite for k8s.gcr.io/pause:3.2 as k8s.gcr.io/pause:3.2 - got k8s.gcr.io/k8s.gcr.io/pause:3.2
    registries_test.go:156: OK  rewrite for busybox as index.docker.io/library/busybox:latest - got index.docker.io/mirrored-library/busybox:latest
    registries_test.go:156: OK  rewrite for busybox as index.docker.io/library/busybox:latest - got index.docker.io/library/busybox/docker:latest
    registries_test.go:156: OK  rewrite for registry.local/team1/images/test as registry.local/team1/images/test:latest - got registry.local/team1-images-test:latest
--- PASS: TestRewrite (0.00s)
=== RUN   TestEndpoints
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm(nil)
        configs: registries.msr(nil)
    registries_test.go:292: Testing image registry.example.com/busybox with:
        mirrors: registries.msm(nil)
        configs: registries.msr(nil)
    registries_test.go:292: Testing image registry.example.com/busybox with:
        mirrors: registries.msm{"registry.example.com":registries.Mirror{Endpoints:[]string{"http://registry.example.com:5000/v2"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr(nil)
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm{"registry.example.com":registries.Mirror{Endpoints:[]string{"https://registry.example.com/v2"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr(nil)
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm{"docker.io":registries.Mirror{Endpoints:[]string{"https://docker.example.com/v2"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr(nil)
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm{"docker.io":registries.Mirror{Endpoints:[]string{"https://docker1.example.com/v2", "https://docker2.example.com/v2"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr(nil)
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm{"*":registries.Mirror{Endpoints:[]string{"https://registry.example.com/v2"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr(nil)
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm{"*":registries.Mirror{Endpoints:[]string{"https://registry.example.com/v2"}, Rewrites:map[string]string(nil)}, "docker.io":registries.Mirror{Endpoints:[]string{"https://docker.example.com/v2"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr(nil)
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm{"docker.io":registries.Mirror{Endpoints:[]string{"https://docker1.example.com/v2", "https://user:bad{@docker2.example.com"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr(nil)
time="2021-12-21T23:46:48-08:00" level=warning msg="Ignoring invalid endpoint URL for registry index.docker.io: parse \"https://user:bad{@docker2.example.com\": net/url: invalid userinfo"
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm{"docker.io":registries.Mirror{Endpoints:[]string{"https://docker1.example.com/v2", "docker2.example.com/v2", "/v2"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr(nil)
time="2021-12-21T23:46:48-08:00" level=warning msg="Ignoring relative endpoint URL for registry index.docker.io: \"docker2.example.com/v2\""
time="2021-12-21T23:46:48-08:00" level=warning msg="Ignoring relative endpoint URL for registry index.docker.io: \"/v2\""
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm(nil)
        configs: registries.msr{"docker.io":registries.RegistryConfig{Auth:(*registries.AuthConfig)(0xc00004af00), TLS:(*registries.TLSConfig)(nil)}}
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm{"docker.io":registries.Mirror{Endpoints:[]string{"https://docker1.example.com/v2"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr{"docker1.example.com":registries.RegistryConfig{Auth:(*registries.AuthConfig)(0xc00004af40), TLS:(*registries.TLSConfig)(nil)}}
    registries_test.go:292: Testing image busybox with:
        mirrors: registries.msm{"docker.io":registries.Mirror{Endpoints:[]string{"http://docker1.example.com:5000/v2"}, Rewrites:map[string]string(nil)}}
        configs: registries.msr{"docker1.example.com:5000":registries.RegistryConfig{Auth:(*registries.AuthConfig)(0xc00004af80), TLS:(*registries.TLSConfig)(nil)}}
--- PASS: TestEndpoints (0.00s)
PASS
ok      github.com/rancher/wharfie/pkg/registries   (cached)
?       github.com/rancher/wharfie/pkg/tarfile  [no test files]
?       github.com/rancher/wharfie/pkg/util [no test files]