gzip: invalid header

[Strandedpirate]

I'm getting "gzip: invalid header" from helm/tiller when trying to install a chart from a helm repository hosted through bitbucket.io. I've also tried an Azure CDN and get the same error.

These are the basic build steps for installing helm and packaging the chart. At the end I simply add the .tgz and updated index to a repository and commit it.

It seems like when the file is fetched/downloaded using helm the gzip compression is stripped off and we are left with a tar file, however if I just use a browser to download the file the gzip compression is left intact and we can install that file local from disk just fine.

What am I doing wrong/How do I fix this?

# install helm
- wget "https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get" -O /usr/local/bin/helm-install.bash
- chmod +x /usr/local/bin/helm-install.bash
- helm-install.bash -v v2.5.1
- helm init --client-only
- export HELM_PACKAGE_VERSION=0.$BITBUCKET_BUILD_NUMBER.0
- helm package chart/design-service --version=$HELM_PACKAGE_VERSION
- helm repo index --url https://buildasign.bitbucket.io/helm --merge siterepo/helm/index.yaml siterepo/helm
...  other steps to check in to repo

helm repo add buildasign https://buildasign.bitbucket.io/helm/ helm repo update helm upgrade -i qa-design-service buildasign/design-service Release "qa-design-service" does not exist. Installing it now. Error: gzip: invalid header

[jascott1]

@Strandedpirate It appears your problem is in the way the chart is (or is not) being gzipped. Debugging the code, it doesn't fail until its actually trying to load the archive from disk saying its not a gzip. Also when I try to open the fetched chart archive (.tgz) in vim, it complains that its not a valid gzipped tar. Not sure what is causing this on your end but it looks like its not gzipped to me. Perhaps to troubleshoot you can package it with helm and then test to see if the chart is gzipped correctly (for me just opening with vim is enough but I have vim.tar compiled in apparently).

[bacongobbler]

I'm going to close this ticket due to inactivity, but please re-open if this still needs to be addressed. Thanks!

[dene14]

@bacongobbler Re-opening... I'm able to reproduce that problem with latest (2.8.1) helm and repo hosted on bitbucket.

So I can confirm that package in the repo is actually gzipped file:

repo/python-0.1.0.tgz: gzip compressed data, extra field, has comment

When I do helm fetch for the repo I'm getting file ~6 times bigger:

$ file python-0.1.0.tgz 
python-0.1.0.tgz: POSIX tar archive

So as we can see, tarball was ungzipped in transit. Same happens if I define direct link to tarball instead of repo.

I suspect that http client that you use in helm isn't properly configured and tries to behave like a browser by unpacking gzipped stuff on the fly.

I can confirm that required headers are set properly serverside:

< Content-Type: application/x-tar
< Content-Encoding: gzip
< Content-Language: en
< Accept-Ranges: bytes
< Content-Length: 5276

[dene14]

@bacongobbler @jascott1 I've just realized that this issue still closed. Could you please reopen?

[jonathan-kosgei]

I have the same issue with a chart hosted on bitbucket.

helm install --name es --debug --dry-run es/
[debug] Created tunnel using local port: '43131'

[debug] SERVER: "localhost:43131"

[debug] Original chart version: ""
[debug] CHART PATH: /home/jonathan/Projects/es-helm/es

Error: error unpacking es-common-0.1.0-pre.0.tgz in es: gzip: invalid header

[bacongobbler]

Is it possible that Bitbucket decompresses gzipped requests? I can't reproduce with chartmuseum or the stable/incubator charts.

[jonathan-kosgei]

Locally I'm able to untar it with tar xvf using the z option gives the error

gzip: stdin: not in gzip format

[bacongobbler]

Can you try a chart not hosted on bitbucket, such as one of the charts from the stable/incubator repos? If so then there's something up with hosting charts on bitbucket specifically.

[itsMicah]

I would also like to share that I am experiencing a similar "Error: gzip: invalid header" error. I am also hosting my charts via Bitbucket.

[rochaporto]

Not using bitbucket but i see the same.

Error: error unpacking sssd-0.1.0.tgz in condor-startd: gzip: invalid header

Trying this manually with wget i get the proper format:

$ wget https://test-charts.web.cern.ch/test-charts/sssd-0.1.0.tgz
$ file sssd-0.1.0.tgz 
sssd-0.1.0.tgz: gzip compressed data, extra field, has comment

While:

$ helm fetch https://test-charts.web.cern.ch/test-charts/sssd-0.1.0.tgz
$ file sssd-0.1.0.tgz 
sssd-0.1.0.tgz: POSIX tar archive

Assuming both fetch and dep update use the same httpgetter then it might need the DisableCompression flag set to True in the http client request? Or an equivalent header set in the request.

[dene14]

D'ouch... I thought I was pretty clear on that case, but it seems not... So I will try to reiterate:

Bitbucket wrongly sets:

Content-Type: application/x-tar
Content-Encoding: gzip

Most probably it happens on their webserver because of ignored MIME table or something like that. In that specific case, for full-featured browsers in order to DOWNLOAD tar.gz file it should be set to something like:

Content-Type: application/x-gzip
Content-Encoding: application/octet-stream 

So this is honestly Bitbucket's problem (or any other misconfigured webserver). However it's only related to a WWW with full featured browsers which have to differentiate rich content, use different behavior for similar data streams in different cases. This manged by all these special headers returned from server.

With helm we have very specific case - we just need to download binary file (and we know it's binary) from server, then untar it with uncompress option and that's it. We should ignore any transformations in the middle. And in our case that transformation in the middle done by HTTP client library used by helm. It tries to behave like full-featured browser, respects headers returned by server and tried to adapt received data for a user, but there is no user behind that - there is a code that tries to uncompress already uncomressed data again.

LMK if I have to provide more details.

[dene14]

@bacongobbler can we please reopen this issue?

[bacongobbler]

Uhh, this issue was never closed to begin with :)

[bacongobbler]

I think @rochaporto is on the right track though. By default we use go's default HTTPClient which as @dene14 suggested, respects headers and adapts the output. In bitbucket's case they send the application/x-tar header when we're requesting a tar.gz. In essence it's an improper server response, but we should be able to just ignore the header as we always expect a tarball to be fetched.

It should be a pretty simple change under NewHTTPGetter to return a http.Client that does not respect the header in the response (and the associated unit test to cover that).

Marking this as a good first issue if someone wants to tackle this!

[bacongobbler]

fixed in #4165

[SychevIgor]

Still the same issue, even when file stored locally

helm version Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

Deploy should be to azure aks cluster. helm install work well. Issue also reproducible from azure devops (vsts) using ubuntu16.04 hosted agent.

Work around: copy helm files to the same directory with helm. In this case - it will work. In Azure DevOps(VSTS) - you may use "Copy" task.

[Elexy]

FWIW I got this inappropriate error as well when I pointed to a yaml file instead or a chart directory.

[mohankumarbm]

Same error when using nexus repo to store helm artifacts.

2018-12-26 18:23:27.813 ERROR 1 --- [.0-8087-exec-10] c.n.s.k.w.e.GenericExceptionHandlers : Internal Server Error java.lang.IllegalStateException: Bake of HelmBakeManifestRequest(namespace=kube-system) failed: Error: gzip: invalid header at com.netflix.spinnaker.rosco.manifests.helm.HelmBakeManifestService.bake(HelmBakeManifestService.java:63) ~[rosco-manifests-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT] at com.netflix.spinnaker.rosco.controllers.V2BakeryController.doBake(V2BakeryController.java:22) ~[rosco-web-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.9.RELEASE.jar:4.3.9.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.9.RELEASE.jar:4.3.9.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)

[RuBiCK]

Same error for local dep charts with helm v2.12.1 and OSX Mojave

If I build the dependencies it creates a gzipped file that can't be decompressed when installing: Error: gzip: invalid header

helm install --dry-run --debug ./Chart.yaml --values values-int.yaml
[debug] Created tunnel using local port: '52958'

[debug] SERVER: "127.0.0.1:52958"

[debug] Original chart version: ""
[debug] CHART PATH: <my_local_directory>/helm/marketplace/Chart.yaml

Error: gzip: invalid header

to simplify, testing with only one dep:

#Requirements.yaml
dependencies:
  - name: redis-ha
    version: 3.0.3
    repository: file://charts/redis-ha

file charts/redis-ha-3.0.3.tgz
redis-ha-3.0.3.tgz: gzip compressed data, extra field, has comment, original size 35840

tar -xvzf charts/redis-ha-3.0.3.tgz works fine

I don't know if it's related but I got an extrange behaviour when updating dependencies:

$ helm dep update
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "local" chart repository (http://127.0.0.1:8879/charts):
    Get http://127.0.0.1:8879/charts/index.yaml: dial tcp 127.0.0.1:8879: connect: connection refused
...Successfully got an update from the "coreos" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Save error occurred:  directory <my_local_directory>/helm/marketplace/charts/redis-ha not found
Deleting newly downloaded charts, restoring pre-update state
Error: directory <my_local_directory>/helm/marketplace/charts/redis-ha not found

But that directory exists and it's readable

$ls -la <my_local_directory>/helm/marketplace/charts/redis-ha
total 40
drwxr-xr-x   6 rub  staff   192  3 ene 09:32 .
drwxr-xr-x   3 rub  staff    96  3 ene 09:44 ..
-rwxr-xr-x   1 rub  staff   479  3 ene 09:26 Chart.yaml
-rwxr-xr-x   1 rub  staff  8256  3 ene 09:26 README.md
drwxr-xr-x  10 rub  staff   320  3 ene 09:32 templates
-rwxr-xr-x   1 rub  staff  3980  3 ene 09:26 values.yaml

[jurgenweber]

I am seeing this problem with helm3.

# helm dep up tenant-api
Error: error unpacking helm-chart.tgz in tenant-api: gzip: invalid header

[jurgenweber]

I have been talking to bitbucket support and we made a feature request; https://jira.atlassian.com/browse/BCLOUD-19694 vote for it!

[bacongobbler]

@jurgenweber I don't believe #4165 was ported over to Helm 3. If you feel like porting the fix over to Helm 3, please feel free!

Re-opening for the moment.

[alanmoment]

I got same error.

My helm repo is in S3 bucket. When I doing helm s3 reindex <repo_name> and I got error below message

helm s3 reindex XXX
traverse the chart repository: load archive from s3 object: gzip: invalid header
Error: plugin "s3" exited with error
make: *** [prepare] Error 1

[jurgenweber]

well you acn fix that in s3, change the metadata, etc.

[sciack]

I hit similar issue with Helm (3.0.3) and Bitbucket, but the problem is this one:

Error: file '<cache>/helm/repository/passwordstore-0.7.2.tgz' does not appear to be a gzipped archive; got 'application/octet-stream'
helm.go:75: [debug] file '<cache>/helm/repository/passwordstore-0.7.2.tgz' does not appear to be a gzipped archive; got 'application/octet-stream'

The gzip is a legit one, but when helm check the content type got application/octet-stream and not x-gzip. I think the problem is in this check in the file chart/loader/archive.go:

if contentType := http.DetectContentType(buffer); contentType != "application/x-gzip" 

That should accept octect/stream too

[andrewnazarov]

We have random does not appear to be a gzipped archive; errors after switching to Helm 3 (3.0.2). Usually with Redis and incubator/raw charts. Subsequent runs work ok, but the error is quite annoying.

At first, it seemed to be a CI runner problem. Due to this error's random nature, it's hard to reproduce and diagnose.

[jurgenweber]

yeah, I have seen that also. Not really related to the OP, should make another issue.

[Ygshvr]

I am facing the same issue. But this is only appearing when one automation is running multiple helm charts together. This is the error that we are getting:

Error: file "/root/.cache/helm/repository/chartA-0.1.0.tgz" does not appear to be a gzipped archive; got "application/octet-stream"

    at ChildProcess.exithandler (child_process.js:281:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:915:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)```

[Shagon94]

According to mimesniff standard application/x-gzip has a byte pattern of 1F 8B 08, checking that manually:

$ file some.tar.gz
some.tar.gz: gzip compressed data, last modified: Tue Apr 21 08:10:11 2020, from Unix, original size modulo 2^32 1054208
$ xxd some.tar.gz | head -1
00000000: 1f8b 0800 63aa 9e5e 0003 ecbb 0558 945d  ....c..^.....X.]

We can see that byte pattern 1f8b 0800; the file in question was created with tar czvf.

the line here https://github.com/helm/helm/blob/b21b00978539ea8270e6b672bc1e6bc07af61475/pkg/chart/loader/archive.go#L88

is properly checking if the content sniffed is of correct mimetype. According to the documentation for the function here:

// DetectContentType implements the algorithm described
// at https://mimesniff.spec.whatwg.org/ to determine the
// Content-Type of the given data. It considers at most the
// first 512 bytes of data. DetectContentType always returns
// a valid MIME type: if it cannot determine a more specific one, it
// returns "application/octet-stream".

So if the underlying method deems it a generic if it can't determine what it is. The check here https://github.com/helm/helm/blob/b21b00978539ea8270e6b672bc1e6bc07af61475/pkg/chart/loader/archive.go#L88-L96

checks if the response is of correct mime type, if not it checks the extension and if both fail then its not a yaml file. The downside of this check is that it doesn't check the file itself, only the extension. A quick way of doing it to check the byte pattern, e.g.:

package main

import (
    "fmt"
    "os"
)

func check(e error) {
    if e != nil {
        panic(e)
    }
}

// 1F 8B 08 is the byte pattern for gzipped files
func checktype(buf []byte) bool {
    return buf[0] == 0x1F && buf[1] == 0x8B && buf[2] == 0x8
}

func main() {
    f, err := os.Open("/tmp/some.tar.gz")
    check(err)

    buffer := make([]byte, 3)
    readBytes, err := f.Read(buffer)
    check(err)
    fmt.Println(readBytes, "bytes read")
    fmt.Println("is zip -", checktype(buffer))
    f.Close()
}

running:

$ go run check.go 
3  bytes read
is zip - true

That way we can confirm if the contents are actually a gzipped file.

[cablespaghetti]

Good detective work @Shagon94. I guess the real question is, why is this intermittent and re-running helps? Sounds like an ordering problem, like the file hasn't been downloaded fully when the check runs.

I thought it might be a problem with the file getting corrupted on download but it is happening too frequently for that I think.

[Shagon94]

Hello @cablespaghetti , if you can provide any urls that show these symptoms or steps to reproduce I can look into it further, so far everything seems to be working properly.

[cablespaghetti]

I'm afraid I've seen only with private s3-based charts so far, using https://github.com/hypnoglow/helm-s3 and when installing multiple instances of the same chart with https://github.com/roboll/helmfile.

My current thinking is that the first call to download the chart begins to download, then the next one starts (in parallel) and because Helm can see a file with the right name in the cache directory it immediately checks the file type of a file which hasn't finished downloading yet.

Thinking aloud, the fix for this may have to be in helmfile but it's odd that it never happened with Helm 2.

[ryanotella]

FWIW, this is still a problem on helm 3.2.0 (so latest ArgoCD is incompatible) but fixed in 3.2.4. It's an ongoing issue with bitbucket.io sites decompressing ".gz" files. It seems to be related to file extension too. Helm charts with ".tgz" fail, but charts with some random extension like ".gzip" download successfully with helm.

[bacongobbler]

Makes sense given it was merged into helm 3.2.2.

[EvanCarroll]

I don't understand this? @bacongobbler did we disable Go's transparent on-the-wire native gzip decompression because of a bug in bitbuket's server implementation?

Someone pasted the link to the problem, and there are tons of reports in the atlassian bug repo over this issue,

• https://jira.atlassian.com/browse/BCLOUD-21555?jql=project%20in%20(BCLOUD%2C%20BSERV)%20AND%20text%20~%20%22gzip%20bitbucket%22%20ORDER%20BY%20cf%5B13230%5D%20ASC

From my understanding, the bug is that Atlassian decompressed on the server (on upload), and rather than re-compressing when they serve or serving as a different mime type they simply serve the decompressed file as if it was compressed. This causes the error observed above. You can see error explained in BCLOUD-21555 the best. Except there the problem pertains to images not helm charts. Also it's not just Helm that's broken on BitBucket, apparently all of Chrome is.

I would not have set DisableCompression to true. The problem is that Atlassian mistakenly marks files they intentionally decompress as compressed. It's in their court.

If I'm right on all this, this is probably the core cause facing the Bitnami project's chart truncation. If this patch was reverted, they could have simply enabled compressed on CloudFlare (because CloudFlare like ~100% of the internet except Atlassian does not mistakenly serve decompressed data as compressed). https://github.com/bitnami/charts/issues/10539

[pdf]

@EvanCarroll that makes no sense - if it was actually decompressed server-side, then setting DisableCompression would have no effect. More likely, that issue you linked is the same thing as what's happening here, just that the user's browser decompressed the file during download, rather than Helm.

[EvanCarroll]

@pdf Right, and and why doesn't decompression work when the browser or when helm does it? Because it's decompressed server side and the mime-type the sever serves it as marks it as still-compressed. Is my read of this wrong? That is, from my read the problem is that they're serving files without compression as being compressed. The error is the result of clients taking them for their word on it.

[pdf]

@EvanCarroll yes, you have it backwards. The file is compressed, the mime type causes it to be decompressed by the client. And it's not just Atlassian, some other hosts do this too, which is why the change was made.

[EvanCarroll]

@pdf ah, I see what you're saying. Let me try again! So while I had it backwards, the problem is essentially that a client that request a .tgz file on a bugged webserver will send it down with a mime-type that instructs the client to unzip on the wire. What they should do is sent it with a mime type as described here. The problem is that this patch which disables compression for everything and affects the index file Bitnami uses which is not uploaded as a compressed .tgz but as a plain yaml file. Only the packages are a .tgz. So the @dene14 assumption,

With helm we have very specific case - we just need to download binary file (and we know it's binary) from server, then untar it with uncompress option and that's it. We should ignore any transformations in the middle. And in our case that transformation in the middle done by HTTP client library used by helm.

Isn't strictly speaking true. Because this patch doesn't just affect the download of binary files, it affects the downloads of package index/yaml too which should allow for on-the-wire decompression so CDNs could turn on transparent compression and solve the Bitnami index problem.

Sorry, did I get it right this time? Kind of embarrassing but I'm trying to wrap my head around it.

[bacongobbler]

I'd consider writing a test case to test your hypothesis. repotest is a test server package you can use to write a test case against a chart repository server.

See also https://pkg.go.dev/net/http/httptest

[jurgenweber]

I sitll get the error even if the encoding is not present;

curl -I https://helm.repo.com/index.yaml
HTTP/2 200
date: Fri, 19 Apr 2024 05:23:03 GMT
content-type: application/octet-stream
content-length: 1780
last-modified: Fri, 19 Apr 2024 05:15:03 GMT
accept-ranges: bytes
etag: "6621fdd7-6f4"
strict-transport-security: max-age=31536000; includeSubDomains
content-type: text/x-yaml

helm pull --repo https://helm.repo.com/ my-service --version 1.0.0
Error: gzip: invalid header

why?

I just put nginx in front of my repo dir, a DiY helm repo.