Closed jordanst3wart closed 1 year ago
I tried to debug the issue, and couldn't figure out why it fails. It seems to me the install script is just a bit flaky. I tried to add the v2 label as well.
The binaries we publish won't work on docker images based on alpine
because we're compiling them against glibc. The sh: aws2: not found
is what happens when you try to execute this binary. If you try it on an image like ubuntu:latest
the installer will work.
Additionally, I don't think we'd ever release an alpine compatible binary like we're doing for our general linux binary. I think we'd be more inclined to just release a docker image that comes with our binary built on alpine.
Thanks for your answer. Is it possible to document that here https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html under "Prerequisites for Linux", or make the error more user friendly in the alpine based image case?
We'll make sure to get this updated in the user guide. I also like the idea of making a more user friendly error message for this case.
@joguSD bummer, alpine is the standard for docker images. Especially with build / ci systems wanting to automate & not wanting to continually download very large docker images.
Can use an official alpine docker image in a multi stage build to copy the binary into another docker image though.
@chadgrant - can you give an example please of your multi-stage dockerfile?
@BarakBD-Globality I do not have one but if the aws-alpine image ever gets created you can just do:
FROM aws_alpine_image:latest as aws
FROM alpine:latest
COPY --from=aws /usr/local/bin/binary /usr/local/bin/binary
Too bad they don't support alpine. It is a standard image.
Additionally, I don't think we'd ever release an alpine compatible binary like we're doing for our general linux binary. I think we'd be more inclined to just release a docker image that comes with our binary built on alpine.
@joguSD has there been any discussion or motion on this? It seems like a pretty big deal considering the prevalence of alpine. People like me use aws
in their cicd and other automation, so if it's not available for alpine-based images, using version 2 is not even an option without significant work! Thanks.
also can't install in busybox
Interestingly if you use this as a base image the AWS cli will install and work properly.
FROM adoptopenjdk/openjdk11:alpine
I was able to do this using the following Dockerfile
FROM adoptopenjdk/openjdk11:alpine
RUN apk add --no-cache \
ca-certificates \
unzip \
curl \
groff \
less \
bash \
openssh-client
WORKDIR /home
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
&& unzip awscliv2.zip \
&& ./aws/install \
&& rm awscliv2.zip \
&& rm -R aws
The reason that the adoptopenjdk image works is because they include the missing glibc libraries. Alphine linux is based on 'musl glibc', a light-weight alternative to a fullblown glibc. The aws cli v2 binaries do not work with musl, they need a few more libraries to work.
Instead of using the Java image, you can also use the Dockerfile below as an example how to run AWS CLI v2 on Alpine Linux:
FROM alpine:3.11
ENV GLIBC_VER=2.31-r0
# install glibc compatibility for alpine
RUN apk --no-cache add \
binutils \
curl \
&& curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
&& curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk \
&& curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \
&& apk add --no-cache \
glibc-${GLIBC_VER}.apk \
glibc-bin-${GLIBC_VER}.apk \
&& curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \
&& unzip awscliv2.zip \
&& aws/install \
&& rm -rf \
awscliv2.zip \
aws \
/usr/local/aws-cli/v2/*/dist/aws_completer \
/usr/local/aws-cli/v2/*/dist/awscli/data/ac.index \
/usr/local/aws-cli/v2/*/dist/awscli/examples \
&& apk --no-cache del \
binutils \
curl \
&& rm glibc-${GLIBC_VER}.apk \
&& rm glibc-bin-${GLIBC_VER}.apk \
&& rm -rf /var/cache/apk/*
The above will download the glibc libraries, download and install AWS CLI v2, and remove some stuff we probably don't need, such as auto-complete. The resulting image is about 100MB in size
So what's the shortest path to get a aws ecr get-login-password … | docker login …
inside a docker container running? (which implies: docker:dind
)?
I'd love to avoid to build my own docker:stable
image with docker:dind
support. At least I could thanks to https://github.com/aws/aws-cli/issues/4685#issuecomment-615872019 solution. Thanks for that!
@bentolor Since docker:dind
also uses alpine, I suspect all you have to do is change the first line of the above mentioned Dockerfile. Instead of FROM alpine:3.11
you should use FROM docker:dind
. It does mean you're building your own image, though....
thanks @blagerweij works really nicely
Also thanks @blagerweij : I grasped the nettle and built a public, custom docker image by copy&pasting your solution: https://hub.docker.com/r/bentolor/docker-dind-awscli
@bentolor I noticed in your repo that your Dockerfile uses FROM docker:stable
instead of FROM docker:dind
. I thought the whole point was to create an image for Docker in Docker with AWS support ?
@blagerweij See the image below from the official Docker image for illustration.
The typical usage is:
docker:dind
instance and thendocker:stable
instance which is linked to the started docker:dind
containerdocker
commands inside the second container which redirects the comamnds via socket connection to docker daemon running inside the first docker:dind
container.Therefore: While it's true that the docker:dind
instance eventually executes your Docker commands, your CLI frontend actually lives in a docker:stable
instance. And that's the place where one i.e. needs to login into AWS ;-)
At least this is the usage pattern I'm aware of (i.e. on using Gitlab CI to build docker images). Maybe there are also direct docker:dind
usage scenarios?
One may use alpine image with glibc from https://hub.docker.com/r/frolvlad/alpine-glibc/ and install aws cli following steps from aws documentation. Worked for me.
@blagerweij I used your script with the docker in docker image. aws cli works fine. I just got an error in build:
/usr/glibc-compat/sbin/ldconfig: /usr/glibc-compat/lib/ld-linux-x86-64.so.2 is not a symbolic link
@amiram Yes I also receive that message, that's a warning that you can simply ignore: It's expecting a symlink, but the file present is a regular file. See https://github.com/envoyproxy/envoy/issues/9078#issuecomment-576837432
Could someone rename this issue to something like: "Distribute binaries for alpine / musl"?
@joguSD @jordanst3wart
Yeah, i'm happy with "Distribute binaries for alpine / musl libc". Let me know if you have any issues
I only need aws s3 cp for a passion project, so I just installed awscli V1 with pip:
FROM alpine:3.12
RUN pip install awscli
RUN aws s3 ..
The reason that the adoptopenjdk image works is because they include the missing glibc libraries. Alphine linux is based on 'musl glibc', a light-weight alternative to a fullblown glibc. The aws cli v2 binaries do not work with musl, they need a few more libraries to work.
Instead of using the Java image, you can also use the Dockerfile below as an example how to run AWS CLI v2 on Alpine Linux:
snip
The above will download the glibc libraries, download and install AWS CLI v2, and remove some stuff we probably don't need, such as auto-complete. The resulting image is about 100MB in size
this adjustment will fetch the latest version automatically if you don't want to hard code the glibc version:
FROM alpine
# install glibc compatibility for alpine
RUN apk --no-cache add \
binutils \
curl \
&& GLIBC_VER=$(curl -s https://api.github.com/repos/sgerrand/alpine-pkg-glibc/releases/latest | grep tag_name | cut -d : -f 2,3 | tr -d \",' ') \
&& curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
&& curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk \
&& curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \
&& apk add --no-cache \
glibc-${GLIBC_VER}.apk \
glibc-bin-${GLIBC_VER}.apk \
&& curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \
&& unzip awscliv2.zip \
&& aws/install \
&& rm -rf \
awscliv2.zip \
aws \
/usr/local/aws-cli/v2/*/dist/aws_completer \
/usr/local/aws-cli/v2/*/dist/awscli/data/ac.index \
/usr/local/aws-cli/v2/*/dist/awscli/examples \
&& apk --no-cache del \
binutils \
curl \
&& rm glibc-${GLIBC_VER}.apk \
&& rm glibc-bin-${GLIBC_VER}.apk \
&& rm -rf /var/cache/apk/*
you can also use the Dockerfile below as an example how to run AWS CLI v2 on Alpine Linux:
The provided Dockerfile/script works but displays this error, which doesn't seem to prevent it from working:
aws/install: line 78: /aws/dist/aws: not found
Strangely, running apk add mandoc
just before aws/install
avoids displaying this error, and deleting that package after installing has no ill effect. Random other packages do not seem to have this protective effect.
Any thoughts?
P.S. apk add less
is required for aws help
to work correctly (it will complain about missing package groff
otherwise).
No need to install glibc to alpine.
FROM alpine:3.13
RUN apk add --no-cache \
acl \
fcgi \
file \
gettext \
git \
curl \
unzip\
python3-dev \
py3-pip \
gcc \
linux-headers \
musl-dev \
libffi-dev \
openssl-dev
RUN git clone --recursive --depth 1 --branch v2 --single-branch https://github.com/aws/aws-cli.git
WORKDIR aws-cli
RUN pip install --ignore-installed -r requirements.txt
RUN pip install -e .
RUN aws --version
Step 7/7 : RUN aws --version ---> Running in be6edcdd97d8 aws-cli/2.1.22 Python/3.8.7 Linux/5.8.0-41-generic source/x86_64.alpine.3 prompt/off
No need to install glibc to alpine.
FROM alpine:3.13 RUN apk add --no-cache \ acl \ fcgi \ file \ gettext \ git \ curl \ unzip\ python3-dev \ py3-pip \ gcc \ linux-headers \ musl-dev \ libffi-dev \ openssl-dev RUN git clone --recursive --depth 1 --branch v2 --single-branch https://github.com/aws/aws-cli.git WORKDIR aws-cli RUN pip install --ignore-installed -r requirements.txt RUN pip install -e . RUN aws --version
Step 7/7 : RUN aws --version ---> Running in be6edcdd97d8 aws-cli/2.1.22 Python/3.8.7 Linux/5.8.0-41-generic source/x86_64.alpine.3 prompt/off
Thanks for that !
anyone has time and able to convert that into a multi-stage building so we get only the necessary librairies and end up with an image without pip3, gcc ...etc ?
Hard to do a multi-stage since aws install script move files on multiple locations. I think it's not a viable solution to copy each files in the long run. Best course of action IMHO is to remove all build packages after install.
Could be in the same RUN to reduce final size.
Multi-stage would be has the advantage that its easier for merging into other images - maybe i have a php script, and to use from php:alpine
but want to use the aws-cli for some reason
Hard to do a multi-stage since aws install script move files on multiple locations. I think it's not a viable solution to copy each files in the long run. Best course of action IMHO is to remove all build packages after install.
Could be in the same RUN to reduce final size.
I gave the "install everything, build, remove as much as I can" a quick try - my image grows by ~300MB. The try is by adding this at the end of Dockerfile: (ie: removing everything, even curl and unzip, except python3 and pip3)
RUN apk del acl \
fcgi \
file \
gettext \
git \
curl \
unzip\
gcc \
linux-headers \
musl-dev \
libffi-dev \
openssl-dev
Pretty sure someone better would get a smaller image, but not sure how small it can get (my hunch says it won't be much smaller ?) Another way would be to create a standalone exe/package - I tried pyinstaller, but coming from no knowledge about it, I did not get anywhere either - maybe someone could try that ?
I'm getting 'ascii' codec can't encode character in position
error when running some awscli commands, I think this is due to unicode stuff. Something I don't get when run awscli on Debian based image.
That might be a different issue @bormansquirrel . I think AWS v2 upgraded python to v3 from v2, which changed a lot of character encoding, and decoding.
Oh yeah, forgot to update it here ...
You would need to generate your locale by installing the glibc-i18n
package as well, as it is indicated in the main documentation of alpine-pkg-glibc
here
@jgournet, here is a multistage. Could more selective I guess. But reduce the size by a factor 2 so far.
FROM alpine:3.13 as builder
RUN apk add --no-cache \
acl \
fcgi \
file \
gettext \
git \
curl \
unzip\
python3-dev \
py3-pip \
gcc \
linux-headers \
musl-dev \
libffi-dev \
openssl-dev
RUN git clone --recursive --depth 1 --branch v2 --single-branch https://github.com/aws/aws-cli.git
WORKDIR aws-cli
RUN pip install --ignore-installed -r requirements.txt
RUN pip install -e .
RUN aws --version
FROM alpine:3.13 as runner
RUN apk add --no-cache python3
COPY --from=builder /usr/bin/aws* /usr/bin/
COPY --from=builder /usr/lib/python3.8/site-packages /usr/lib/python3.8/site-packages
COPY --from=builder /aws-cli /aws-cli
@goshiz aws help
doesn't work without groff
, so i added apk add --no-cache groff
ref: https://github.com/aws/aws-cli/issues/1957#issuecomment-687455928 https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html
Here is the dockerfile:
FROM alpine:3.13 as builder
RUN apk add --no-cache \
acl \
fcgi \
file \
gettext \
git \
curl \
unzip\
python3-dev \
py3-pip \
gcc \
linux-headers \
musl-dev \
libffi-dev \
openssl-dev
RUN git clone --recursive --depth 1 --branch v2 --single-branch https://github.com/aws/aws-cli.git
WORKDIR aws-cli
RUN pip install --ignore-installed -r requirements.txt
RUN pip install -e .
RUN aws --version
FROM alpine:3.13 as runner
RUN apk add --no-cache python3
COPY --from=builder /usr/bin/aws* /usr/bin/
COPY --from=builder /usr/lib/python3.8/site-packages /usr/lib/python3.8/site-packages
COPY --from=builder /aws-cli /aws-cli
I find it odd that the binary produced is still dependent on python3. :confused: I was trying to do:
docker cp runner:/usr/bin/aws /tmp/aws
and uplift the binary.
I published a binary here: https://github.com/jordanst3wart/awsv2-musl-binary/releases/tag/0.0.1 I don't know if it would work on different systems.
Another attempt with PyInstaller
to avoid python3 dependency:
FROM python:3-alpine3.13 AS installer
RUN apk add --no-cache \
acl \
binutils \
curl \
fcgi \
file \
g++ \
gcc \
gettext \
git \
libc-dev \
libffi-dev \
linux-headers \
musl-dev \
openssl-dev \
pwgen \
py3-pip \
python3-dev \
unzip \
zlib-dev
RUN git clone --recursive --depth 1 --branch v2 --single-branch https://github.com/aws/aws-cli.git
WORKDIR aws-cli
# Follow https://github.com/six8/pyinstaller-alpine to install pyinstaller on alpine
RUN pip install --upgrade pip \
&& pip install pycrypto \
&& git clone --depth 1 --single-branch --branch v$(grep PyInstaller requirements-build.txt | cut -d'=' -f3) https://github.com/pyinstaller/pyinstaller.git /tmp/pyinstaller \
&& cd /tmp/pyinstaller/bootloader \
&& CFLAGS="-Wno-stringop-overflow -Wno-stringop-truncation" python ./waf configure --no-lsb all \
&& pip install .. \
&& rm -Rf /tmp/pyinstaller \
&& cd -
RUN scripts/installers/make-exe
RUN unzip dist/awscli-exe.zip && \
./aws/install --bin-dir /aws-cli-bin/
FROM alpine:3.13
RUN apk --no-cache add groff
COPY --from=installer /usr/local/aws-cli/ /usr/local/aws-cli/
COPY --from=installer /aws-cli-bin/ /usr/local/bin/
Additionally, I don't think we'd ever release an alpine compatible binary like we're doing for our general linux binary. I think we'd be more inclined to just release a docker image that comes with our binary built on alpine.
@joguSD has there been any discussion or motion on this? It seems like a pretty big deal considering the prevalence of alpine. People like me use
aws
in their cicd and other automation, so if it's not available for alpine-based images, using version 2 is not even an option without significant work! Thanks.
Sorry if my wording was confusing. There is absolutely value in providing an alpine docker image that includes the AWS CLI V2, that if possible does not include glibc.
In my previous comment I was stating that I think we would rather provide an alpine based docker image with the AWS CLI V2 pre-installed rather than pre-built binaries that are compatible with alpine.
I think we would rather provide an alpine based docker image with the AWS CLI V2 pre-installed rather than pre-built binaries that are compatible with alpine.
@joguSD this would make it harder to use for us as we already use other base images it would be hard to step out of. E.g., if we're using a node-alpine' image, we can't just run Docker in Docker. We'd have to refactor our image to use a
dind` base or to use your base. That's just not much of a starter for us to revamp whole automated pipelines and image building that is already well-established. Not to mention the local developer experience for quick tests of things during development time.
Providing the docker build instructions would be far more valuable to us than an image, because then we can just copy and paste them into our existing image-generation infrastructure.
Based off @erks PyInstaller
solution, this can also install a specific version (not just the latest), by building and installing the associated botocore version.
FROM python:3-alpine3.13 AS installer
ENV AWSCLI_VERSION=2.1.36
RUN apk add --no-cache \
gcc \
git \
libc-dev \
libffi-dev \
openssl-dev \
py3-pip \
zlib-dev
RUN git clone --recursive --depth 1 --branch ${AWSCLI_VERSION} --single-branch https://github.com/aws/aws-cli.git
WORKDIR /aws-cli
# Follow https://github.com/six8/pyinstaller-alpine to install pyinstaller on alpine
RUN pip install --no-cache-dir --upgrade pip \
&& pip install --no-cache-dir pycrypto \
&& git clone --depth 1 --single-branch --branch v$(grep PyInstaller requirements-build.txt | cut -d'=' -f3) https://github.com/pyinstaller/pyinstaller.git /tmp/pyinstaller \
&& cd /tmp/pyinstaller/bootloader \
&& CFLAGS="-Wno-stringop-overflow -Wno-stringop-truncation" python ./waf configure --no-lsb all \
&& pip install .. \
&& rm -Rf /tmp/pyinstaller \
&& cd - \
&& boto_ver=$(grep botocore setup.cfg | cut -d'=' -f3) \
&& git clone --single-branch --branch v2 https://github.com/boto/botocore /tmp/botocore \
&& cd /tmp/botocore \
&& git checkout $(git log --grep $boto_ver --pretty=format:"%h") \
&& pip install . \
&& rm -Rf /tmp/botocore \
&& cd -
RUN sed -i '/botocore/d' requirements.txt \
&& scripts/installers/make-exe
RUN unzip dist/awscli-exe.zip && \
./aws/install --bin-dir /aws-cli-bin
FROM alpine:3.13
RUN apk --no-cache add groff
COPY --from=installer /usr/local/aws-cli/ /usr/local/aws-cli/
COPY --from=installer /aws-cli-bin/ /usr/local/bin/
Another option: AWS can contribute a v2 version of this Alpine package aws-cli to the testing release track. Then we don't need to futz around with multi-stage builds for every container we want to run AWS CLI v2 from, the result would be smaller and more stable, and neither AWS nor its customers would need to do any more support.
Great: aws-cli introduced the use of awscrt in version 2.2.0 ... which does not install on alpine :( a few random logs of the failure, for those searching in google:
Building wheels for collected packages: awscrt
Building wheel for awscrt (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-127zvgks/awscrt_4c6af7b8b13645ffb0d04ecbf328de3d/setup.py'"'"'; __file__='"'"'/tmp/pip-install-127zvgks/awscrt_4c6af7b8b13645ffb0d04ecbf328de3d/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-1ciqi1om
cwd: /tmp/pip-install-127zvgks/awscrt_4c6af7b8b13645ffb0d04ecbf328de3d/
Complete output (56 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/awscrt
copying awscrt/common.py -> build/lib.linux-x86_64-3.9/awscrt
copying awscrt/s3.py -> build/lib.linux-x86_64-3.9/awscrt
copying awscrt/__init__.py -> build/lib.linux-x86_64-3.9/awscrt
copying awscrt/http.py -> build/lib.linux-x86_64-3.9/awscrt
copying awscrt/mqtt.py -> build/lib.linux-x86_64-3.9/awscrt
copying awscrt/_test.py -> build/lib.linux-x86_64-3.9/awscrt
copying awscrt/auth.py -> build/lib.linux-x86_64-3.9/awscrt
copying awscrt/io.py -> build/lib.linux-x86_64-3.9/awscrt
copying awscrt/exceptions.py -> build/lib.linux-x86_64-3.9/awscrt
copying awscrt/crypto.py -> build/lib.linux-x86_64-3.9/awscrt
creating build/lib.linux-x86_64-3.9/awscrt/eventstream
copying awscrt/eventstream/__init__.py -> build/lib.linux-x86_64-3.9/awscrt/eventstream
copying awscrt/eventstream/rpc.py -> build/lib.linux-x86_64-3.9/awscrt/eventstream
running build_ext
--- Building dependency: aws-lc (RelWithDebInfo) ---
/usr/bin/cmake -DCMAKE_PREFIX_PATH=/tmp/pip-install-127zvgks/awscrt_4c6af7b8b13645ffb0d04ecbf328de3d/build/deps/install -DCMAKE_INSTALL_PREFIX=/tmp/pip-install-127zvgks/awscrt_4c6af7b8b13645ffb0d04ecbf328de3d/build/deps/install -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_TESTING=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_LIBSSL=OFF /tmp/pip-install-127zvgks/awscrt_4c6af7b8b13645ffb0d04ecbf328de3d/crt/aws-lc
CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles". CMAKE_MAKE_PROGRAM is not set. You probably need to select a different build tool.
-- Configuring incomplete, errors occurred!
See also "/tmp/pip-install-127zvgks/awscrt_4c6af7b8b13645ffb0d04ecbf328de3d/build/deps/aws-lc/CMakeFiles/CMakeOutput.log".
Updated previous PyInstaller solution https://github.com/aws/aws-cli/issues/4685#issuecomment-820136291 . Installing both make
and cmake
seems to get past that awscrt error.
FROM python:3-alpine3.13 AS installer
ENV AWSCLI_VERSION=2.2.0
RUN apk add --no-cache \
gcc \
git \
libc-dev \
libffi-dev \
openssl-dev \
py3-pip \
zlib-dev \
make \
cmake
RUN git clone --recursive --depth 1 --branch ${AWSCLI_VERSION} --single-branch https://github.com/aws/aws-cli.git
WORKDIR /aws-cli
# Follow https://github.com/six8/pyinstaller-alpine to install pyinstaller on alpine
RUN pip install --no-cache-dir --upgrade pip \
&& pip install --no-cache-dir pycrypto \
&& git clone --depth 1 --single-branch --branch v$(grep PyInstaller requirements-build.txt | cut -d'=' -f3) https://github.com/pyinstaller/pyinstaller.git /tmp/pyinstaller \
&& cd /tmp/pyinstaller/bootloader \
&& CFLAGS="-Wno-stringop-overflow -Wno-stringop-truncation" python ./waf configure --no-lsb all \
&& pip install .. \
&& rm -Rf /tmp/pyinstaller \
&& cd - \
&& boto_ver=$(grep botocore setup.cfg | cut -d'=' -f3) \
&& git clone --single-branch --branch v2 https://github.com/boto/botocore /tmp/botocore \
&& cd /tmp/botocore \
&& git checkout $(git log --grep $boto_ver --pretty=format:"%h") \
&& pip install . \
&& rm -Rf /tmp/botocore \
&& cd -
RUN sed -i '/botocore/d' requirements.txt \
&& scripts/installers/make-exe
RUN unzip dist/awscli-exe.zip \
&& ./aws/install --bin-dir /aws-cli-bin
FROM alpine:3.13
RUN apk --no-cache add groff
COPY --from=installer /usr/local/aws-cli/ /usr/local/aws-cli/
COPY --from=installer /aws-cli-bin/ /usr/local/bin/
Just checking in. Even though Alpine is perfectly capable of running the latest versions of Python, it's still not compatible with the wrapper binary used with v2.
Can someone from AWS explain why the binary wrapper was introduced with v2? My wild guess is that it was intended to solve issues for people who needed to use the CLI, but didn't know jack about Python, and the pre-compiled binary was a way to mitigate those issues.
Either way, it would be helpful to understand why AWS doesn't also just release the source-code as a pip package or whatever so that the rest of us can get our work done.
(Could be solved by #5578)
The reason that the adoptopenjdk image works is because they include the missing glibc libraries. Alphine linux is based on 'musl glibc', a light-weight alternative to a fullblown glibc. The aws cli v2 binaries do not work with musl, they need a few more libraries to work.
Instead of using the Java image, you can also use the Dockerfile below as an example how to run AWS CLI v2 on Alpine Linux:
FROM alpine:3.11 ENV GLIBC_VER=2.31-r0 # install glibc compatibility for alpine RUN apk --no-cache add \ binutils \ curl \ && curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \ && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk \ && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \ && apk add --no-cache \ glibc-${GLIBC_VER}.apk \ glibc-bin-${GLIBC_VER}.apk \ && curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \ && unzip awscliv2.zip \ && aws/install \ && rm -rf \ awscliv2.zip \ aws \ /usr/local/aws-cli/v2/*/dist/aws_completer \ /usr/local/aws-cli/v2/*/dist/awscli/data/ac.index \ /usr/local/aws-cli/v2/*/dist/awscli/examples \ && apk --no-cache del \ binutils \ curl \ && rm glibc-${GLIBC_VER}.apk \ && rm glibc-bin-${GLIBC_VER}.apk \ && rm -rf /var/cache/apk/*
The above will download the glibc libraries, download and install AWS CLI v2, and remove some stuff we probably don't need, such as auto-complete. The resulting image is about 100MB in size
@vsimon I am able to install was, but how do I log in to AWS in my Dockerfile without exposing my secrets.
This is the error I get while trying to download my .env from s3 download failed: s3://admin-***/***.env to - Unable to locate credentials
Hi @skyzyx,
See #6186 for discussion about releasing a source distribution, which indeed could aid in solving this.
@tobslob,
Discussions for getting credentials into a Dockerfile
is out of scope for this repository - I would try Stack Overflow, as the approach may change depending on the environment you're running in (Kubernetes, a CI/CD environment, etc.).
If you want to know how to get credentials into a running container, that documentation is here:
https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-docker.html#cliv2-docker-share-files
The recent source distribution proposal shows how you can build the AWS CLI v2 under Alpine Linux now, using a Docker container as an example:
https://github.com/kyleknap/aws-cli/blob/source-proposal/proposals/source-install.md#alpine-linux
Third parties have also indicated success with Alpine 3.14:
https://github.com/aws/aws-cli/pull/6352#issuecomment-909083091
Please note the limited scope of this proposal:
https://github.com/kyleknap/aws-cli/blob/source-proposal/proposals/source-install.md#non-goals
While this proposal helps distribution maintainers make the AWS CLI v2 available through standard package managers, it does not address adding official support for installing the AWS CLI v2 through them. It is focused on providing a mechanism for users that want or need to install the AWS CLI v2 from source.
We would graciously accept any feedback on the PR that the Alpine community has on this proposal, and assist in making sure it is working as intended.
Thanks so much for the Alpine source build example! This is great news.
I think the next step in order to provide feedback on the PR would be to build a package on Alpine. That way we can find and address any remaining gotchas.
Does anyone know if any Alpine maintainers/packagers have this on their radar? If not, I can start learning how to package/release for Alpine. It's been a few years since I was a package maintainer, but it's just like falling off a bike, right? You never forget how much it hurts... :wink:
edit started working on Alpine packages. There's some bugs in the builds for the dependencies, and working through the other issues found on the PR.
I think building the packages should be doable, but building them in a way that is conformant to Alpine's packaging standards may present more work, but is also doable. Basically I'm having to hack around/patch the CMakeLists.txt
for each aws package. And if aws-lc
is a requirement, that package will probably need to change its default file locations, and so more patches for the packages that depend on it (which is most of 'em). Ah, distro packaging......
@pwillis-els said:
edit started working on Alpine packages. There's some bugs in the builds for the dependencies, and working through the other issues found on the PR.
I'm also beginning to poke at this. My pipeline is different (I'm creating synchronized builds across macOS, Windows, AL2, CentOS 7, Ubuntu 20.04, and Alpine 3.14) so I'm using nfpm instead of APKBUILD
to keep the multi-platform build pipeline simple. As I make progress, I'm happy to share notes on dependencies and compilation steps for Alpine.
I've been using the example provided some time ago, but it is now failing on the latest python (3.10) alpine release. Anyone would be able to know why ? so we can open a ticket with the relevant party...
Dockerfile used:
#FROM python:3.9-alpine3.13 # <- this works fine
FROM python:3-alpine # <- this does not work
RUN apk upgrade --no-cache && \
apk add --no-cache \
acl \
binutils \
cmake \
curl \
fcgi \
file \
g++ \
gcc \
gettext \
git \
libc-dev \
libffi-dev \
linux-headers \
make \
musl-dev \
openssl-dev \
pwgen \
py3-pip \
python3-dev \
unzip \
zlib-dev
RUN git clone --recursive --depth 1 --branch v2 --single-branch https://github.com/aws/aws-cli.git
WORKDIR aws-cli
# Follow https://github.com/six8/pyinstaller-alpine to install pyinstaller on alpine
RUN pip install --upgrade pip \
&& pip install pycrypto \
&& git clone --depth 1 --single-branch --branch v$(grep PyInstaller requirements-build.txt | cut -d'=' -f3) https://github.com/pyinstaller/pyinstaller.git /tmp/pyinstaller \
&& cd /tmp/pyinstaller/bootloader \
&& CFLAGS="-Wno-stringop-overflow -Wno-stringop-truncation" python ./waf configure --no-lsb all \
&& pip install .. \
&& rm -Rf /tmp/pyinstaller \
&& cd -
RUN scripts/installers/make-exe
RUN unzip dist/awscli-exe.zip && \
./aws/install --bin-dir /aws-cli-bin/
Can't paste the error message as it's a few pages long
Errors with: