Harbor UI does not show helm value file and throws error "unknown: gzip: invalid header"

[mcoreix]

We are also still facing this issue. Is there anything we can do about this?

Harbor Version: v2.10.0 Helm Version: v3.14.3 Cosign Version: v2.2.2

2024-04-09T07:51:01Z [ERROR] [/controller/event/handler/internal/artifact.go:259]: scan artifact t527/helm-charts/mailhog@sha256:2aed7314c345b8a7f071e4c8257b797e287ed8a24c67224d5bfd9bc8e0a5a6a1 failed, error: the configured scanner Trivy does not support scanning artifact with mime type application/vnd.oci.image.manifest.v1+json

{"errors":[{"code":"UNKNOWN","message":"unknown: gzip: invalid header"}]}

harbor-core-9db585df4-clk92 core 2024-04-04T07:26:44Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: gzip: invalid header"}]}
harbor-core-9db585df4-2wnc2 core 2024-04-04T07:26:51Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: gzip: invalid header"}]}
harbor-core-9db585df4-2wnc2 core 2024-04-04T07:26:54Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: gzip: invalid header"}]}
harbor-core-9db585df4-x785z core 2024-04-04T07:26:57Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: gzip: invalid header"}]}
harbor-core-9db585df4-clk92 core 2024-04-04T07:26:58Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: gzip: invalid header"}]}
harbor-core-9db585df4-2wnc2 core 2024-04-04T07:26:59Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: gzip: invalid header"}]}

To reproduce:

helm create test-upload
helm dependency update test-upload
helm dependency build test-upload
echo ${HELM_CHART_SIGNING_KEY_PASSWORD} | helm package --sign --key 'My Signing Key' --keyring ~/.gnupg/secring.gpg --passphrase-file - test-upload --version 0.0.1 --app-version v1.0.0
helm push test-upload-0.0.1.tgz oci://harbor.your.domain/your_project/helm-charts 2>&1 | tee push.log
DIGEST=$(cat push.log | grep 'Digest:' | awk '{ print $2 }')
cosign sign --tlog-upload=false --key your_cosign.key harbor.your.domain/your_project/helm-charts/test-upload@${DIGEST}

Dev console in browser also shows readme.md with status code 500 Internal Server Error. Strange thing is, some charts that are uploaded equal don't show any errors.

[zyyw]

@mcoreix could you please share with us the output of this command:

crane manifest harbor.your.domain/your_project/helm-charts/test-upload@${DIGEST}

Because based on my investigation:

helm create test-upload
helm dependency update test-upload
helm dependency build test-upload
helm package ./test-upload/ --version 0.0.1 --app-version v1.0.0
helm push test-upload-0.0.1.tgz oci://<harbor-endpoint>/library/
> output:
> Pushed: <harbor-endpoint>/library/test-upload:0.0.1
> Digest: sha256:c66806c9f8661a1d3a7c28cc398933c6c8f90d5c972bf31003854340030ae6b2

And when I run crane manifest <harbor-endpoint>/library/test-upload:0.0.1 | jq, output like this:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "sha256:9c652d3b7360d822fa40c2f94891beb831974a6c8defd0f7f97da5daf11c4261",
    "size": 145
  },
  "layers": [
    {
      "mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip",
      "digest": "sha256:8cdf8991227727dee4be2b793114c860c8ee96e4ae85ef4886c0d4e253a3b227",
      "size": 3757
    }
  ]
}

From the output of the crane manifest, we can see that the config.mediaType is application/vnd.cncf.helm.config.v1+json, which explains that why it is not supported to scan with trivy.

[zyyw]

As for Dev console in browser also shows readme.md with status code 500 Internal Server Error, tried to this:

helm push harbor-1.14.2.tgz oci://<harbor-endpoint>/library

the README.md displays as expected. (harbor-endpoint is in harbor version 2.10.0)

[mcoreix]

@mcoreix could you please share with us the output of this command:

crane manifest harbor.your.domain/your_project/helm-charts/test-upload@${DIGEST}

Because based on my investigation:

helm create test-upload
helm dependency update test-upload
helm dependency build test-upload
helm package ./test-upload/ --version 0.0.1 --app-version v1.0.0
helm push test-upload-0.0.1.tgz oci://<harbor-endpoint>/library/
> output:
> Pushed: <harbor-endpoint>/library/test-upload:0.0.1
> Digest: sha256:c66806c9f8661a1d3a7c28cc398933c6c8f90d5c972bf31003854340030ae6b2

And when I run crane manifest <harbor-endpoint>/library/test-upload:0.0.1 | jq, output like this:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "sha256:9c652d3b7360d822fa40c2f94891beb831974a6c8defd0f7f97da5daf11c4261",
    "size": 145
  },
  "layers": [
    {
      "mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip",
      "digest": "sha256:8cdf8991227727dee4be2b793114c860c8ee96e4ae85ef4886c0d4e253a3b227",
      "size": 3757
    }
  ]
}

From the output of the crane manifest, we can see that the config.mediaType is application/vnd.cncf.helm.config.v1+json, which explains that why it is not supported to scan with trivy.

Here i get this error:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "sha256:dc3d04819898a9256dcf004c3df5002e069b94b6219b4e8c2c4df883e678757d",
    "size": 149
  },
  "layers": [
    {
      "mediaType": "application/vnd.cncf.helm.chart.provenance.v1.prov",
      "digest": "sha256:267ded90ba903e0befdc0351488f82701c8554cd3ebcb4b90b1efdd868ff2895",
      "size": 1089
    },
    {
      "mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip",
      "digest": "sha256:53361efb6c657ee8d910c2f838b08d99242ec22d645e820843758624004f4802",
      "size": 112934
    }
  ],
  "annotations": {
    "org.opencontainers.image.created": "2024-04-15T03:00:47Z",
    "org.opencontainers.image.description": "A Helm chart for Kubernetes",
    "org.opencontainers.image.title": "mailhog",
    "org.opencontainers.image.version": "1.0.1-17-6"
  }
}

Here i dont get the error:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "sha256:1a281b4c71f69d289f8ff5990308f854b4b46a04356bb53b018ce6bf393b7d6a",
    "size": 149
  },
  "layers": [
    {
      "mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip",
      "digest": "sha256:c5330796fe167ed5332a460b75abaf4dedf26e35bd4dcc12eb13b9f684e41cb2",
      "size": 113647
    },
    {
      "mediaType": "application/vnd.cncf.helm.chart.provenance.v1.prov",
      "digest": "sha256:c8ecaee05cacf2b9e742619473c43627e21c1e7ac35563d1279f6e4630e98cba",
      "size": 1089
    }
  ],
  "annotations": {
    "org.opencontainers.image.created": "2024-04-09T08:45:56Z",
    "org.opencontainers.image.description": "A Helm chart for Kubernetes",
    "org.opencontainers.image.title": "mailhog",
    "org.opencontainers.image.version": "1.0.1-17-5"
  }
}

Same Chart...but different Tags

[zyyw]

could you please share the core.log when you encounter with this error:

Dev console in browser also shows readme.md with status code 500 Internal Server Error.

[mcoreix]

@zyyw 2024-04-17T11:34:14Z [INFO] [/controller/registry/controller.go:222]: Start regular health check for registries with interval 5m0s 2024-04-17T11:40:28Z [ERROR] [/lib/http/error.go:57]: {"errors":[{"code":"UNKNOWN","message":"unknown: gzip: invalid header"}]}

[mcoreix]

@zyyw any updates?

[mcoreix]

@zyyw any updates?

[zyyw]

Hi @mcoreix could you please upload the two oci compatible helm charts (one that works fine, and the other that errors out) to somewhere, maybe dockerhub, so that I can access and try to reproduce the scenario you mentioned above with them on a Harbor instance.

Thanks

[oweis-nordlb]

@zyyw it can be any helm chart. I can upload one chart, get the error and upload the same chart again without an error

[0Styless]

@zyyw Is there anything new about this issue?