search

GoogleCloudPlatform / berglas

A tool for managing secrets on Google Cloud
https://cloud.google.com/secret-manager
Apache License 2.0
1.24k stars 96 forks source link

Downloaded binary is sometimes broken. #174

Closed civitaspo closed 2 years ago

civitaspo commented 2 years ago

Hi maintainers,

Since about the day before yesterday, the berglas binary downloaded from this download link (https://storage.googleapis.com/berglas/main/linux_amd64/berglas) is sometimes corrupted. It doesn't mean that it is broken every time. I investigated and found that the binary is gzip compressed. Could you please investigate this problem?

sethvargo commented 2 years ago

Can you tell me how you're downloading the binary? If you request gzip, the response will be gzipped.

shio-phys commented 2 years ago

I have same problem, when using curl to download the binary. Here is the log of the curl with the --verbose option.

$ curl --verbose -o berglas https://storage.googleapis.com/berglas/main/linux_amd64/berglas
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2404:6800:4004:822::2010...
* TCP_NODELAY set
* Connected to storage.googleapis.com (2404:6800:4004:822::2010) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [236 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [96 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4423 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [114 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.storage.googleapis.com
*  start date: Nov  1 02:51:46 2021 GMT
*  expire date: Jan 24 02:51:45 2022 GMT
*  subjectAltName: host "storage.googleapis.com" matched cert's "*.googleapis.com"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7ff63f811c00)
> GET /berglas/main/linux_amd64/berglas HTTP/2
> Host: storage.googleapis.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< x-guploader-uploadid: ADPycdtpBqfujPAtPvsWp1ZsX37gk4m4lj0iv7t4d0t4IAJfso15miUlM9DRdLqhNBJuS7wWYIwPkJyz2feWx1ZcX5Otx2_PAA
< expires: Fri, 19 Nov 2021 01:13:14 GMT
< date: Fri, 19 Nov 2021 00:13:14 GMT
< last-modified: Mon, 08 Nov 2021 22:36:49 GMT
< etag: "82f47e94c0f4bb5e77b738ce547189c6"
< x-goog-generation: 1636411009818469
< x-goog-metageneration: 1
< x-goog-stored-content-encoding: gzip
< x-goog-stored-content-length: 4763832
< content-type: application/octet-stream
< content-encoding: gzip
< x-goog-hash: crc32c=bI0HQw==
< x-goog-hash: md5=gvR+lMD0u153tzjOVHGJxg==
< x-goog-storage-class: MULTI_REGIONAL
< accept-ranges: bytes
< vary: Accept-Encoding
< content-length: 4763832
< server: UploadServer
< cache-control: public, max-age=3600
< age: 0
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<
{ [1178 bytes data]
100 4652k  100 4652k    0     0  11.1M      0 --:--:-- --:--:-- --:--:-- 11.0M
* Connection #0 to host storage.googleapis.com left intact
* Closing connection 0

$ file berglas
berglas: gzip compressed data, from Unix, original size modulo 2^32 13504512
civitaspo commented 2 years ago

Can you tell me how you're downloading the binary? If you request gzip, the response will be gzipped.

The following log shows that gzip compression is being applied randomly.

for _ in {1..30}; do 
    date
    curl -s -o berglas https://storage.googleapis.com/berglas/main/linux_amd64/berglas
    file berglas
    rm -vf berglas
    echo
done

Fri Nov 19 09:59:25 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 09:59:29 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 09:59:33 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 09:59:38 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 09:59:42 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 09:59:46 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 09:59:50 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 09:59:54 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 09:59:59 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:03 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:07 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:12 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:16 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:21 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:25 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:29 JST 2021
berglas: gzip compressed data, from Unix, original size modulo 2^32 13504512
removed 'berglas'

Fri Nov 19 10:00:30 JST 2021
berglas: gzip compressed data, from Unix, original size modulo 2^32 13504512
removed 'berglas'

Fri Nov 19 10:00:32 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:36 JST 2021
berglas: gzip compressed data, from Unix, original size modulo 2^32 13504512
removed 'berglas'

Fri Nov 19 10:00:38 JST 2021
berglas: gzip compressed data, from Unix, original size modulo 2^32 13504512
removed 'berglas'

Fri Nov 19 10:00:40 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:44 JST 2021
berglas: gzip compressed data, from Unix, original size modulo 2^32 13504512
removed 'berglas'

Fri Nov 19 10:00:46 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:50 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:00:55 JST 2021
berglas: gzip compressed data, from Unix, original size modulo 2^32 13504512
removed 'berglas'

Fri Nov 19 10:00:56 JST 2021
berglas: gzip compressed data, from Unix, original size modulo 2^32 13504512
removed 'berglas'

Fri Nov 19 10:00:58 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:01:02 JST 2021
berglas: gzip compressed data, from Unix, original size modulo 2^32 13504512
removed 'berglas'

Fri Nov 19 10:01:04 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'

Fri Nov 19 10:01:08 JST 2021
berglas: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=lZCl3MFTMUprwygrmtYJ/rtfQKeK-Tm27g7bTFQng/DR9fxw5VcgO1CW3OppdV/RfrS-LNwiHeKrfvNWYgm, stripped
removed 'berglas'
sethvargo commented 2 years ago

Hi folks - I'm trying to track this down.

frankyn commented 2 years ago

Hey folks, checking in, are you still seeing this issue? We reverted a change that introduced this behavior and following-up.

Thank you for your patience

sethvargo commented 2 years ago

I'm no longer seeing the issue. Since it's a US holiday weekend, I'm going to wait until Monday and then close this out.