search

zarf-dev / zarf

DevSecOps for Air Gap & Limited-Connection Systems. https://zarf.dev/
Apache License 2.0
1.37k stars 165 forks source link

Images not being found by 'zarf prepare find-images' #585

Closed TheFutonEng closed 2 years ago

TheFutonEng commented 2 years ago

Environment

Device and OS: Mac App version: v0.19.4 Kubernetes distro being used: N/A Other:

Steps to reproduce

  1. Try to find images in the Istio Control Plane repo on Repo1

Expected result

Images are found

Actual Result

No images are returned

Visual Proof (screenshots, videos, text, etc)

[rmengert@Robs-MacBook-Pro:~/projects/hncd_demo_7-5/zarf-pkg-vultron-demo]
$ zarf prepare find-images -p /chart
  ✔  Processing helm chart https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-controlplane.git@1.13.2-bb.1                                                                                 
  ✔  Templating helm chart dummy                                                                                                                                                                    
[rmengert@Robs-MacBook-Pro:~/projects/hncd_demo_7-5/zarf-pkg-vultron-demo]
$

Severity/Priority

Minor I guess?

Additional Context

The Istio Control Plane values file does not have images at the root layer of the file.

One image can be found at {{ .Values.cni.image }} and another at {{ .Values.postInstallHook }}. Not sure if this is the nature of the bug or just because these images have weird keys associated with them.

jeff-mccoy commented 2 years ago

Could you share the zarf package yaml or one that reproduces this? The way that Zarf seeks out images is by first trying to helm template + run manifests / kustomizations through the K8s API to find actual image refs in a podspec, it then does a flat text fuzzy search to try and find additional "OCI path-like images": (?mi)"([a-z0-9\-./]+:[\w][\w.\-]{0,127})" and then tests matches to see if they are actual image paths.

TheFutonEng commented 2 years ago

@jeff-mccoy, surely see below.

kind: ZarfPackageConfig
metadata:
  name: big-bang-core-demo
  description: "Deploy Big Bang Core - HNCD"
  # Big Bang / Iron Bank are only amd64
  architecture: amd64

components:
  # - name: flux
  #   required: true
  #   manifests:
  #     - name: flux-installer
  #       kustomizations:
  #         - https://repo1.dso.mil/platform-one/big-bang/bigbang.git//base/flux?ref=1.36.0

  #   images:
  #     - registry1.dso.mil/ironbank/fluxcd/helm-controller:v0.21.0
  #     - registry1.dso.mil/ironbank/fluxcd/kustomize-controller:v0.25.0
  #     - registry1.dso.mil/ironbank/fluxcd/notification-controller:v0.23.5
  #     - registry1.dso.mil/ironbank/fluxcd/source-controller:v0.24.4

  - name: bigbang
    description: "Git repositories and OCI images used by Big Bang Core"
    required: true
    repos:
      # - https://repo1.dso.mil/platform-one/big-bang/bigbang.git@1.36.0
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-controlplane.git@1.13.2-bb.1
      # - https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-operator.git@1.13.2-bb.1
      # - https://repo1.dso.mil/platform-one/big-bang/apps/core/jaeger.git@2.30.0-bb.2
      # - https://repo1.dso.mil/platform-one/big-bang/apps/core/kiali.git@1.51.0-bb.0
      # - https://repo1.dso.mil/platform-one/big-bang/apps/core/cluster-auditor.git@1.4.0-bb.4
      # - https://repo1.dso.mil/platform-one/big-bang/apps/core/policy.git@3.8.1-bb.0
      # - https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring.git@35.2.0-bb.3
      # - https://repo1.dso.mil/platform-one/big-bang/apps/core/elasticsearch-kibana.git@0.7.1-bb.0
      # - https://repo1.dso.mil/platform-one/big-bang/apps/core/eck-operator.git@2.0.0-bb.2
      # - https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit.git@0.20.0-bb.1
    images:
      - registry1.dso.mil/ironbank/big-bang/grafana/grafana-plugins:8.6.2
      - registry1.dso.mil/ironbank/bigbang/cluster-auditor/opa-exporter:v0.0.4
      - registry1.dso.mil/ironbank/elastic/eck-operator/eck-operator:2.0.0
      - registry1.dso.mil/ironbank/elastic/elasticsearch/elasticsearch:7.17.1
      - registry1.dso.mil/ironbank/elastic/kibana/kibana:7.17.1
      - registry1.dso.mil/ironbank/kiwigrid/k8s-sidecar:1.18.0
      - registry1.dso.mil/ironbank/opensource/fluent/fluent-bit:1.9.3
      - registry1.dso.mil/ironbank/opensource/istio/operator:1.13.2
      - registry1.dso.mil/ironbank/opensource/jaegertracing/all-in-one:1.33.0
      - registry1.dso.mil/ironbank/opensource/jaegertracing/jaeger-collector:1.33.0
      - registry1.dso.mil/ironbank/opensource/jaegertracing/jaeger-operator:1.33.0
      - registry1.dso.mil/ironbank/opensource/jaegertracing/jaeger-query:1.33.0
      - registry1.dso.mil/ironbank/opensource/kiali/kiali-operator:v1.51.0
      - registry1.dso.mil/ironbank/opensource/kubernetes/kube-state-metrics:v2.4.2
      - registry1.dso.mil/ironbank/opensource/openpolicyagent/gatekeeper:v3.8.1
      - registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-operator:v0.56.2
      - registry1.dso.mil/ironbank/opensource/prometheus/alertmanager:v0.24.0
      - registry1.dso.mil/ironbank/opensource/prometheus/node-exporter:v1.3.1
      - registry1.dso.mil/ironbank/opensource/prometheus/prometheus:v2.35.0
    manifests:
      - name: big-bang-config
        kustomizations:
          - "kustomization/bigbang"

Please let me know if you need anything else! This is Rob M by the way.

andrewg-xyz commented 2 years ago

To give a bit more information, when running with zarf v0.19.4 These images are not discovered by zarf prepare find-images -p /chart or zarf prepare find-images -p chart

      - registry1.dso.mil/ironbank/opensource/istio/install-cni:1.13.2
      - registry1.dso.mil/ironbank/big-bang/base:1.0.0
      - registry1.dso.mil/ironbank/opensource/istio/pilot:1.13.2
      - registry1.dso.mil/ironbank/opensource/istio/proxyv2:1.13.2
      - registry1.dso.mil/ironbank/opensource/ingress-nginx/kube-webhook-certgen:v1.1.1
      - registry1.dso.mil/ironbank/big-bang/base:1.17.0
      - registry1.dso.mil/ironbank/opensource/prometheus/alertmanager:v0.24.0
      - registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-config-reloader:v0.56.2
      - registry1.dso.mil/ironbank/opensource/kiali/kiali:v1.51.0

When these repos are specified in the component:

      - https://repo1.dso.mil/platform-one/big-bang/bigbang.git@1.36.0
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-operator.git@1.13.2-bb.1
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/istio-controlplane.git@1.13.2-bb.1
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring.git@35.2.0-bb.3
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/elasticsearch-kibana.git@0.7.1-bb.0
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/eck-operator.git@2.0.0-bb.2
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit.git@0.20.0-bb.1
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/jaeger.git@2.30.0-bb.2
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/kiali.git@1.51.0-bb.0
      - https://repo1.dso.mil/platform-one/big-bang/apps/core/cluster-auditor.git@1.4.0-bb.4
jeff-mccoy commented 2 years ago

How did you determine these images were required by Big Bang core? By looking at the s3 release artifacts or something else?

RothAndrew commented 2 years ago

The package-images.yaml file in each Big Bang release is actually quite good these days.

jeff-mccoy commented 2 years ago

Yeah the problem is, it's not in the repo and without navigating the Gitlab API (and knowing that the repo is actually hosted on gitlab), we have no way from Zarf to attempt to parse that. They also do have extra images, depending on your configuration, e.g.: https://umbrella-bigbang-releases.s3-us-gov-west-1.amazonaws.com/packages/istio-operator/1.13.4-bb.0/images.txt

jeff-mccoy commented 2 years ago

We could add the release images.txt to files and have Zarf read that, but that feels like just doing overkill vs just making a lightweight script (that I'm sure someone already has) to produce the list for you.

jeff-mccoy commented 2 years ago

Since this isn't a bug with the way zarf find images, but a limitation of how operators can obfuscate images, going to close this issue.

RothAndrew commented 2 years ago 
wget https://umbrella-bigbang-releases.s3-us-gov-west-1.amazonaws.com/umbrella/1.39.0/package-images.yaml
yq '.package-image-list.*.images' package-images.yml | yq 'unique'

🤯

results in:

- "registry1.dso.mil/ironbank/big-bang/base:2.0.0"
- "registry1.dso.mil/ironbank/opensource/istio/pilot:1.13.5"
- "registry1.dso.mil/ironbank/opensource/istio/proxyv2:1.13.5"
- "registry1.dso.mil/ironbank/opensource/istio/install-cni:1.13.5"
- "registry1.dso.mil/ironbank/tetrate/istio/istioctl:1.13.5-tetratefips-v0"
- "registry1.dso.mil/ironbank/tetrate/istio/proxyv2:1.13.5-tetratefips-v0"
- "registry1.dso.mil/ironbank/tetrate/istio/pilot:1.13.5-tetratefips-v0"
- "registry1.dso.mil/ironbank/tetrate/istio/install-cni:1.13.5-tetratefips-v0"
- "registry1.dso.mil/ironbank/opensource/istio/operator:1.13.5"
- "registry1.dso.mil/ironbank/tetrate/istio/operator:1.13.5-tetratefips-v0"
- "registry1.dso.mil/ironbank/opensource/ingress-nginx/kube-webhook-certgen:v1.1.1"
- "registry1.dso.mil/ironbank/opensource/jaegertracing/all-in-one:1.35.2"
- "registry1.dso.mil/ironbank/opensource/jaegertracing/jaeger-operator:1.35.0"
- "registry1.dso.mil/ironbank/opensource/kiali/kiali-operator:v1.51.0"
- "registry1.dso.mil/ironbank/opensource/kiali/kiali:v1.51.0"
- "registry1.dso.mil/ironbank/bigbang/cluster-auditor/opa-exporter:v0.0.4"
- "registry1.dso.mil/ironbank/opensource/kubernetes/kubectl:v1.22.2"
- "registry1.dso.mil/ironbank/opensource/openpolicyagent/gatekeeper:v3.8.1"
- "registry1.dso.mil/ironbank/elastic/elasticsearch/elasticsearch:8.2.3"
- "registry1.dso.mil/ironbank/elastic/kibana/kibana:8.2.3"
- "registry1.dso.mil/ironbank/elastic/eck-operator/eck-operator:2.3.0"
- "registry1.dso.mil/ironbank/opensource/fluent/fluent-bit:1.9.6"
- "registry1.dso.mil/ironbank/big-bang/grafana/grafana-plugins:9.0.1"
- "registry1.dso.mil/ironbank/kiwigrid/k8s-sidecar:1.19.2"
- "registry1.dso.mil/ironbank/opensource/kubernetes/kube-state-metrics:v2.5.0"
- "registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-config-reloader:v0.57.0"
- "registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-operator:v0.57.0"
- "registry1.dso.mil/ironbank/opensource/prometheus/alertmanager:v0.24.0"
- "registry1.dso.mil/ironbank/opensource/prometheus/node-exporter:v1.3.1"
- "registry1.dso.mil/ironbank/opensource/prometheus/prometheus:v2.36.2"
- "registry1.dso.mil/ironbank/twistlock/console/console:22.06.179"
- "registry1.dso.mil/ironbank/twistlock/defender/defender:22.06.179"
- "registry1.dso.mil/ironbank/big-bang/argocd:v2.4.4"
- "registry1.dso.mil/ironbank/bitnami/redis:7.0.0-debian-10-r3"
- "registry1.dso.mil/ironbank/opensource/dexidp/dex:v2.30.3"
- "registry1.dso.mil/ironbank/istio-ecosystem/authservice:0.5.1"
- "registry1.dso.mil/ironbank/opensource/minio/operator:v4.4.25"
- "registry1.dso.mil/ironbank/opensource/minio/console:v0.19.0"
- "registry1.dso.mil/ironbank/opensource/minio/minio:RELEASE.2022-04-30T22-23-53Z"
- "registry1.dso.mil/ironbank/bitnami/analytics/redis-exporter:1.37.0"
- "registry1.dso.mil/ironbank/gitlab/gitlab/alpine-certificates:15.1.2"
- "registry1.dso.mil/ironbank/gitlab/gitlab/cfssl-self-sign:1.4.1"
- "registry1.dso.mil/ironbank/gitlab/gitlab/gitaly:15.1.2"
- "registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-container-registry:15.1.2"
- "registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-shell:15.1.2"
- "registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-sidekiq:15.1.2"
- "registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-toolbox:15.1.2"
- "registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-webservice:15.1.2"
- "registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-workhorse:15.1.2"
- "registry1.dso.mil/ironbank/gitlab/gitlab/kubectl:15.1.2"
- "registry1.dso.mil/ironbank/opensource/minio/mc:RELEASE.2022-07-06T14-54-36Z"
- "registry1.dso.mil/ironbank/opensource/minio/minio:RELEASE.2022-07-08T00-05-23Z"
- "registry1.dso.mil/ironbank/opensource/postgres/postgresql12:12.11"
- "registry1.dso.mil/ironbank/redhat/ubi/ubi8:8.6"
- "registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-exporter:15.1.2"
- "registry1.dso.mil/ironbank/gitlab/gitlab-runner/gitlab-runner:v15.0.0"
- "registry1.dso.mil/ironbank/gitlab/gitlab-runner/gitlab-runner-helper:v15.0.0"
- "registry1.dso.mil/ironbank/sonatype/nexus/nexus:3.40.1-01"
- "registry1.dso.mil/ironbank/big-bang/sonarqube:8.9.9-community"
- "registry1.dso.mil/ironbank/anchore/engine/engine:1.1.0"
- "registry1.dso.mil/ironbank/anchore/enterprise/enterprise:4.0.2"
- "registry1.dso.mil/ironbank/anchore/enterpriseui/enterpriseui:4.0.0"
- "registry1.dso.mil/ironbank/opensource/mattermost/mattermost-operator:v1.18.1"
- "registry1.dso.mil/ironbank/opensource/mattermost/mattermost:7.0.1"
- "registry1.dso.mil/ironbank/opensource/minio/mc:RELEASE.2022-02-26T03-58-31Z"
- "registry1.dso.mil/ironbank/opensource/minio/minio:RELEASE.2022-02-26T02-54-46Z"
- "registry1.dso.mil/ironbank/opensource/postgres/postgresql11:11.10"
- "registry1.dso.mil/ironbank/opensource/postgres/postgresql12:12.8"
- "registry1.dso.mil/ironbank/opensource/velero/velero-plugin-for-aws:v1.5.0"
- "registry1.dso.mil/ironbank/opensource/velero/velero:v1.9.0"
- "registry1.dso.mil/ironbank/opensource/velero/velero-plugin-for-csi:v0.3.0"
- "registry1.dso.mil/ironbank/opensource/velero/velero-plugin-for-microsoft-azure:v1.5.0"
- "registry.dso.mil/platform-one/big-bang/apps/security-tools/keycloak/keycloak-ib:18.0.2-1.2.0-1"
jeff-mccoy commented 2 years ago

Yeah I used that for the BB 1.39 update. The actual long-term answer for bb I think is including the images in the helm chart definition.