search

myENA / consul-backinator

Command line Consul backup and restore utility supporting KVs, ACLs and Queries
Mozilla Public License 2.0
226 stars 22 forks source link

Release archive extract from Dockerfile does not work #47

Closed bygui86 closed 3 years ago

bygui86 commented 3 years ago

Hi guys

I tried to add consul-backinator to a custom Docker image I'm building.

Here the Dockerfile

# base image
FROM python:3.8.8-slim

# install additional packages
RUN apt-get update
RUN apt-get install -y curl ca-certificates tzdata unzip

# set current dir
WORKDIR /usr/bin

# install consul-backinator
RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.tar.gz
RUN tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz
RUN chmod +x ./consul-backinator

# ...

I have no idea why, but running the command tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz gives me this error:

...
 => [10/17] RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.tar.gz                                                                                                                                                                                                                               0.5s
 => ERROR [11/17] RUN tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz                                                                                                                                                                                                                                                                                            0.3s
------
 > [11/17] RUN tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz:
#14 0.236
#14 0.236 gzip: stdin: not in gzip format
#14 0.236 tar: Child returned status 1
#14 0.236 tar: Error is not recoverable: exiting now
------
executor failed running [/bin/sh -c tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz]: exit code: 2

I tried with zip archive 

# ...

# install consul-backinator
RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.zip
RUN unzip consul-backinator-1.6.6-amd64-linux.zip
RUN chmod +x ./consul-backinator

# ...

but more or less the message is the same (command unzip consul-backinator-1.6.6-amd64-linux.zip):

...
 => [10/17] RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.zip
 => ERROR [11/17] RUN unzip consul-backinator-1.6.6-amd64-linux.zip                                                                                                                                                                                                                                                                                                  0.3s
------
 > [11/17] RUN unzip consul-backinator-1.6.6-amd64-linux.zip:
#15 0.225   End-of-central-directory signature not found.  Either this file is not
#15 0.225   a zipfile, or it constitutes one disk of a multi-part archive.  In the
#15 0.225   latter case the central directory and zipfile comment will be found on
#15 0.225   the last disk(s) of this archive.
#15 0.225 unzip:  cannot find zipfile directory in one of consul-backinator-1.6.6-amd64-linux.zip or
#15 0.225         consul-backinator-1.6.6-amd64-linux.zip.zip, and cannot find consul-backinator-1.6.6-amd64-linux.zip.ZIP, period.
#15 0.225 Archive:  consul-backinator-1.6.6-amd64-linux.zip
------
executor failed running [/bin/sh -c unzip consul-backinator-1.6.6-amd64-linux.zip]: exit code: 9

I tried to download and extract packages locally (macos) and the errors are the same!

It seems that archives are corrupted. Any idea why? How can I solve this?

aaronhurt commented 3 years ago

Downloaded locally file looks good:

[aarhurt@mbp ~/Downloads]$ file consul-backinator-1.6.6-amd64-*
consul-backinator-1.6.6-amd64-linux.tar.gz:  gzip compressed data, last modified: Wed Dec  9 01:48:15 2020, from Unix, original size modulo 2^32 32270336
consul-backinator-1.6.6-amd64-linux.zip:     Zip archive data, at least v1.0 to extract

My SHA256 sums also match, can you check yours?

[aarhurt@mbp ~/Downloads]$ shasum -a256 consul-backinator-1.6.6-amd64-linux.zip
9bf044a13aa7be2c5c25712535dcecbfde8f1f3e347f6e6a15cd42834a41560f  consul-backinator-1.6.6-amd64-linux.zip
[aarhurt@mbp ~/Downloads]$ cat consul-backinator-1.6.6-SHA256SUMS|grep 9bf044a13aa7be2c5c25712535dcecbfde8f1f3e347f6e6a15cd42834a41560f
9bf044a13aa7be2c5c25712535dcecbfde8f1f3e347f6e6a15cd42834a41560f  consul-backinator-1.6.6-amd64-linux.zip
[aarhurt@mbp ~/Downloads]$ shasum -a256 consul-backinator-1.6.6-amd64-linux.tar.gz
d9f2517c678904b75be6b0aff86cfd06375970380d6ae75353df7dafc38f191f  consul-backinator-1.6.6-amd64-linux.tar.gz
[aarhurt@mbp ~/Downloads]$ cat consul-backinator-1.6.6-SHA256SUMS|grep d9f2517c678904b75be6b0aff86cfd06375970380d6ae75353df7dafc38f191f
d9f2517c678904b75be6b0aff86cfd06375970380d6ae75353df7dafc38f191f  consul-backinator-1.6.6-amd64-linux.tar.gz

The files also extract fine locally. What utillity are you using inside your image?

[aarhurt@mbp ~/Downloads]$ tar tzvf consul-backinator-1.6.6-amd64-linux.tar.gz
drwxr-xr-x  0 njohnson staff       0 Dec  8 19:48 ./consul-backinator-1.6.6-amd64-linux/
-rwxr-xr-x  0 njohnson staff 32268288 Dec  8 19:48 ./consul-backinator-1.6.6-amd64-linux/consul-backinator-1.6.6
[aarhurt@mbp ~/Downloads]$ unzip -l consul-backinator-1.6.6-amd64-linux.zip
Archive:  consul-backinator-1.6.6-amd64-linux.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  12-08-2020 19:48   consul-backinator-1.6.6-amd64-linux/
 32268288  12-08-2020 19:48   consul-backinator-1.6.6-amd64-linux/consul-backinator-1.6.6
---------                     -------
 32268288                     2 files
bygui86 commented 3 years ago

Here the command to download (both locally and inside Dockerfile): curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.tar.gz

Here the command to extract the archive (both locally and inside Dockerfile): tar tzvf consul-backinator-1.6.6-amd64-linux.tar.gz

Here the output from tar tzvf consul-backinator-1.6.6-amd64-linux.tar.gz:

tar: Error opening archive: Unrecognized archive format

Here the output from file consul-*:

consul-backinator-1.6.6-SHA256SUMS:         HTML document text, ASCII text, with very long lines, with no line terminators
consul-backinator-1.6.6-amd64-linux.tar.gz: HTML document text, ASCII text, with very long lines, with no line terminators

Here the output from shasum -a256 consul-backinator-1.6.6-amd64-linux.tar.gz:

5e2b345cefb4deec12dca962e2b69d3c52c5fa45ea05f70beedbb442ee090928  consul-backinator-1.6.6-amd64-linux.tar.gz

No output from cat consul-backinator-1.6.6-SHA256SUMS | grep 5e2b345cefb4deec12dca962e2b69d3c52c5fa45ea05f70beedbb442ee090928

As you can see file type is not gzip as expected but HTML and sha256 does not match.

I tried to download another tar.gz package curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-334.0.0-linux-x86_64.tar.gz and everything works perfectly.

I always use curl -O, I don't understand why in this case it does not work...

FYI: running on MacOS Big Sur 11.2.3

aaronhurt commented 3 years ago

Ahh, I see. It's how GitHub does downloads. They do a redirect. Sorry I didn't catch that before.

[aarhurt@mbp ~/Downloads]$ curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   648  100   648    0     0   1732      0 --:--:-- --:--:-- --:--:--  1732
[aarhurt@mbp ~/Downloads]$ file consul-backinator-1.6.6-amd64-linux.tar.gz
consul-backinator-1.6.6-amd64-linux.tar.gz: HTML document text, ASCII text, with very long lines, with no line terminators
[aarhurt@mbp ~/Downloads]$ cat consul-backinator-1.6.6-amd64-linux.tar.gz
<html><body>You are being <a href="https://github-releases.githubusercontent.com/43525589/af0df980-398e-11eb-9736-9930e8f0b7d9?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210401%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20210401T160333Z&amp;X-Amz-Expires=300&amp;X-Amz-Signature=3ab36bbc0c7096d6cd55a382ba5437b9f88b1a2a7f55b92febf0283d74a13bf7&amp;X-Amz-SignedHeaders=host&amp;actor_id=0&amp;key_id=0&amp;repo_id=43525589&amp;response-content-disposition=attachment%3B%20filename%3Dconsul-backinator-1.6.6-amd64-linux.tar.gz&amp;response-content-type=application%2Foctet-stream">redirected</a>.</body></html>

Use the -L flag to curl so it will follow redirects:

[aarhurt@mbp ~/Downloads]$ rm consul-backinator-1.6.6-amd64-linux.tar.gz
[aarhurt@mbp ~/Downloads]$ curl -L -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   648  100   648    0     0   4050      0 --:--:-- --:--:-- --:--:--  4050
100 7823k  100 7823k    0     0  12.1M      0 --:--:-- --:--:-- --:--:-- 12.1M
[aarhurt@mbp ~/Downloads]$ file consul-backinator-1.6.6-amd64-linux.tar.gz
consul-backinator-1.6.6-amd64-linux.tar.gz: gzip compressed data, last modified: Wed Dec  9 01:48:15 2020, from Unix, original size modulo 2^32 32270336
aaronhurt commented 3 years ago

@bygui86 Going to close this issue. Let me know if the above solution doesn't resolve the issue.

bygui86 commented 3 years ago

@leprechau thanks a lot for that! Yeah the issue is solved! I was not considering the redirect!! So stupid :(