Publish V2 as a Ubuntu PPA or Ubuntu Snap

[GauntletWizard]

Is your feature request related to a problem? Please describe. The current recommended instructions for installing the AWS CLI V2 involve downloading a zip file, unzipping it, and installing it. This seems to be limited, lacking features like bash-completion that are included with the ubuntu-provided awscli package.

Describe the solution you'd like A package on Ubuntu's PPA, or other Debian package repository, that provides an awscli package and python3-botocore package compatible with the existing awscli packages in the ubuntu repositories.

Describe alternatives you've considered PyPi support has been officially discontinued, and the official docker image provides a number of problems around privileges and credentials communication.

Additional context AWSCLI v1 PPA Apparently unofficial AWSCLI v1 Snap - Well out of date, currently aws-cli/1.15.58

[kdaily]

Hi @GauntletWizard, thanks for the feature request.

Per this comment (https://github.com/aws/aws-cli/issues/4947#issuecomment-793192340), after a source distribution is made available, support for distribution methods will be discussed with the community.

I'm going to leave this issue open and mark the requests for v1 snap as duplicates and close them - I imagine going forward a CLI V1 Snap would not be prioritized.

[talregev]

I didn't understand why you close my issues and my PRs. #7177

I successful compile aws-cli 2 on snap package for arm. I can share it with you, also make a package here on the repo. Also make it for the ci.

[talregev]

I will put my PRs to reference to whom who want to build the snap package. https://github.com/aws/aws-cli/pull/7179 https://github.com/aws/aws-cli/pull/7180

[zekena2]

Hey in this comment It''s said "In Q2, we plan to begin work on a source bundle that is well documented, easily ingestible by package managers" it's been 19 months now since this comment. Is there any progress in that sense? I don't see many package managers that's able to install v2.

[sqrammi]

Insane that Amazon is still distributing zip files and expecting people to run scripts as root to install their software.

Where is the PPA?

[brlin-tw]

expecting people to run scripts as root to install their software.

Technically packages from PPA also run in-package scripts as root to do the installation.

[sqrammi]

@brlin-tw Yes, but they are signed and validated by the OS seamlessly. Heck, I'd even be happy with a snap package that doesn't require root to install software at all.

[brlin-tw]

@brlin-tw Yes, but they are signed and validated by the OS seamlessly.

Fair point.

[lizthegrey]

The Snap now exists at https://snapcraft.io/aws-cli in beta channel, see sudo snap install aws-cli --channel=v2/beta

However, it has a bug in its access permissions, it needs to ask for permissions to ~/.aws/cli/cache not just to ~/.aws, otherwise reading/writing the STS assumerole credentials cache fails.

[giner]

Good news! Thank you.

In the meanwhile (until the issue is fixed) the snap could be used in devmode

sudo snap install aws-cli --channel=v2/beta --devmode

[marrek-az]

I uploaded revisions 177 (ARM) and 178 that will hopefully address the permissions issue in v2.

[lizthegrey]

I uploaded revisions 177 (ARM) and 178 that will hopefully address the permissions issue in v2.

Confirmed working!

[pdecat]

New 178 version is working fine on Ubuntu 22.04, but I can't figure how to get bash completion for the Snap.

Is it working for anyone? :pray:

[marrek-az]

We are working on the tab completion issue, and that should be fixed soon. We are also looking into additional issues resulting from strict confinement, and working out the best way to address those. I will update again once we have decided on a course of action.

[lizthegrey]

For those that a snap won't work for, I have https://launchpad.net/~lizthegrey/+archive/ubuntu/misc built with jammy backports of the lunar packages and I do refresh it periodically.

[pdecat]

Thank you so much @lizthegrey! And completion does work out of the box with that package :tada:

[talregev]

@lizthegrey Can you also create aws-cli v2 snap into armhf architecture?

[talregev]

@lizthegrey There is a bug. I am using aws-cli inside snapcraft build. It install, but when I run it, it have runtime error: on core22. It mean ubuntu 22.04 on x64 architecture.

    build-snaps:
    - aws-cli/v2/beta

:: + aws-cli.aws --version
:: /snap/snapd/x1/usr/lib/snapd/snap-confine: error while loading shared libraries: libudev.so.1: failed to map segment from shared object

[talregev]

If someone want to compile it for arm:

sudo apt install libffi-dev python3-pip python3-dev musl-dev rustc cargo libssl-dev asn1c libsystemd-dev cmake -y
mkdir -p $HOME/.cargo 
mount -t tmpfs none $HOME/.cargo
pip install git+https://github.com/aws/aws-cli@2.9.19

[lizthegrey]

If someone want to compile it for arm:

sudo apt install libffi-dev python3-pip python3-dev musl-dev rustc cargo libssl-dev asn1c libsystemd-dev cmake -y
mkdir -p $HOME/.cargo 
mount -t tmpfs none $HOME/.cargo
pip install git+https://github.com/aws/aws-cli@2.9.19

Sure I can force an armhf build for PPA (I don't control the snap).

[talregev]

If someone want to compile it for arm:

sudo apt install libffi-dev python3-pip python3-dev musl-dev rustc cargo libssl-dev asn1c libsystemd-dev cmake -y
mkdir -p $HOME/.cargo 
mount -t tmpfs none $HOME/.cargo
pip install git+https://github.com/aws/aws-cli@2.9.19

Sure I can force an armhf build for PPA (I don't control the snap).

That will be helpful too!

[marrek-az]

The snap is available for the AWS Graviton ARM architecture. I will look at the libudev.so.1 issue.

[talregev]

The snap is available for the AWS Graviton ARM architecture. I will look at the libudev.so.1 issue.

What do you mean by Graviton ARM architecture? As I see in the https://snapcraft.io/aws-cli Only amd64 and arm64. No armhf

[marrek-az]

The ARM64 snap will install on Graviton instances. At present, we're not set up to build armhf packages. I can discuss that with the team, but there are a few other snap-related issues I need to address first.

[talregev]

The ARM64 snap will install on Graviton instances. At present, we're not set up to build armhf packages. I can discuss that with the team, but there are a few other snap-related issues I need to address first.

Thank you!

[marrek-az]

@lizthegrey There is a bug. I am using aws-cli inside snapcraft build. It install, but when I run it, it have runtime error: on core22. It mean ubuntu 22.04 on x64 architecture.

    build-snaps:
    - aws-cli/v2/beta

:: + aws-cli.aws --version
:: /snap/snapd/x1/usr/lib/snapd/snap-confine: error while loading shared libraries: libudev.so.1: failed to map segment from shared object

I am unable to reproduce this. I created a simple snap and included the aws-cli in build-snaps as indicated. I then ran snapcraft build --shell --use-lxd and executed aws-cli.aws --version in the shell. Instead of the error, I see: aws-cli/2.9.19 Python/3.8.10 Linux/5.15.0-1028-aws source/x86_64.ubuntu-core.20 prompt/off. I tested this with core20 and core22, but couldn't reproduce the error.

[talregev]

I did it in github action inside docker:

        docker run \
          --rm \
          --tty \
          --privileged \
          --volume $PWD:/root \
          --workdir /root \
          --platform linux/amd64 \
          diddledani/snapcraft:core22 \
          "snap run snapcraft --verbosity verbose pack --destructive-mode --output tal.snap"

[talregev]

@lizthegrey Can you update your ppa and add armhf without wait for the next update?

[lizthegrey]

@lizthegrey Can you update your ppa and add armhf without wait for the next update?

yup working on it, just, it's a spare time project, I need to do another upload with a bumped ppa version number since I can't re-upload the same version number.

[talregev]

@lizthegrey Can you update your ppa and add armhf without wait for the next update?

yup working on it, just, it's a spare time project, I need to do another upload with a bumped ppa version number since I can't re-upload the same version number.

Thank you!

[talregev]

@marrek-az Did you succeeded to reproduce? I try some of solutions from here and I didn't succeeded: https://stackoverflow.com/questions/13502156/what-are-possible-causes-of-failed-to-map-segment-from-shared-object-operation

Also try to add to my snap build and it didn't work:

    build-packages:
    - libudev1

[marrek-az]

@marrek-az Did you succeeded to reproduce? I try some of solutions from here and I didn't succeeded: https://stackoverflow.com/questions/13502156/what-are-possible-causes-of-failed-to-map-segment-from-shared-object-operation

Also try to add to my snap build and it didn't work:

    build-packages:
    - libudev1

I have not been able to make the attempt. I appreciate the extra information you were able to provide. It remains in my queue to address.

[marrek-az]

Sorry for the delay. I have updated the v2/beta packages. There have been some extensive changes. We were unable to support common customer use cases with a strict confinement snap at this time, so I have rebuilt the snap using classic confinement.

To update, you need to manually execute sudo snap refresh --channel=v2/beta --classic because of the change to the confinement model.

Tab completion and other issues still not addressed. I will get to them as quickly as possible.

[mathieujobin]

Actually looking at downgrading from v2, which is now shipped by Ubuntu 23.04

get-login-password token is expiring too quickly... unusable!

I removed the apt package and installed via snap, although I dislike snap

$ snap list
Name               Version          Rev    Tracking       Publisher   Notes
aws-cli            1.27.51          176    latest/beta    aws✓        -

[marrek-az]

@mathieujobin are you seeing the token expire

Actually looking at downgrading from v2, which is now shipped by Ubuntu 23.04

get-login-password token is expiring too quickly... unusable!

Are you seeing this behavior (token expiration) with v2 that is not installed from snap? If so, please open an issue against the cli instead of commenting on this issue.

[mathieujobin]

OK, Thanks @marrek-az

created issue #7977

[dannysauer]

Snaps are great if you never want to use a file in /tmp and you like to spend a year or more tracking down weird security issues in a Snap which perpetually stays in beta... How many years does it take to just generate a Debian/CONTROL file for a real package, though?

[talregev]

@dannysauer You should open a PR and create a debian package in aws github action ci.

[mathieujobin]

Ain't the apt package already present?

$ sudo apt search awscli
Sorting... Done
Full Text Search... Done
awscli/mantic,mantic 2.12.0-1 all
  Unified command line interface to Amazon Web Services

I use snap because I am stuck with v1 due to an auth bug with v2.

$ snap list 
Name               Version          Rev    Tracking       Publisher   Notes
aws-cli            1.27.51          176    latest/beta    aws✓        -

with v2

$ login succesful && docker push

denied: Your authorization token has expired. Reauthenticate and try again.

[marrek-az]

Snaps are great if you never want to use a file in /tmp and you like to spend a year or more tracking down weird security issues in a Snap which perpetually stays in beta... How many years does it take to just generate a Debian/CONTROL file for a real package, though?

@dannysauer

The process for updating a package in the Ubuntu repository to update a package after release will not allow the package to iterate as quickly as the aws-cli package needs to. Staying current is easier with the snap. Debian package updates are tracked through https://launchpad.net/ubuntu/+source/awscli

The aws-cli snap uses classic confinement, which should allow you to create files in /tmp or wherever.

[marrek-az]

I use snap because I am stuck with v1 due to an auth bug with v2.

$ snap list 
Name               Version          Rev    Tracking       Publisher   Notes
aws-cli            1.27.51          176    latest/beta    aws✓        -

with v2

$ login succesful && docker push

denied: Your authorization token has expired. Reauthenticate and try again.

@mathieujobin I would suggest you switch to the v1/stable channel if you need to remain on v1 (as opposed to "latest"). As for the bug, I would be interested in whether it exists in the latest version of the snap, if it does not exist in an unsnapped installation.

[mathieujobin]

Thank you @marrek-az

I can confirm, latest v2 from snap works fine. only the package from apt appears broken. at least for Ubuntu 23.04 and 23.10, I can't quite remember 22.04 or 22.10 but its been a while I have this bug. So thank you

It might be a good idea to push an update before the next LTS, 24.04

[mathieujobin]

@marrek-az I am still having the auth token expiry problem with v2 via snap...

using aws-cli v2 with ecr get-login-password
WARNING! Your password will be stored unencrypted in /home/mathieu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
The push refers to repository [ACCTID.dkr.ecr.us-east-1.amazonaws.com/appname-staging]
5f70bf18a086: Preparing 
f67c5cafe5fd: Preparing 
57d109d7dde9: Preparing 
7b3cf558db85: Preparing 
db8b36e78beb: Preparing 
2803895576b0: Waiting 
30d416c2183d: Waiting 
b6e7ce9ac328: Waiting 
b5dcba89a352: Waiting 
5e972fa97790: Waiting 
799aa655a46e: Waiting 
b59209a73413: Waiting 
66fb5be9a007: Waiting 
7b98130eeda0: Waiting 
d409b9bd1388: Waiting 
bc778503ecd9: Waiting 
69acdd69a66a: Waiting 
6762f41357f4: Waiting 
e03520a78b06: Waiting 
a7b4a0269643: Waiting 
b8a64a777371: Waiting 
04f9728bf48f: Waiting 
5458e810c3d3: Waiting 
ec17bbe8d56e: Waiting 
00ad22cbda7d: Waiting 
5ecef6fc4bba: Waiting 
b388b37f60f3: Waiting 
2345c3d7b10e: Waiting 
5f70bf18a086: Preparing 
54c97f2c0579: Waiting 
ded1dc99d7cc: Waiting 
036bb3c92734: Waiting 
5bb14cfff0ea: Waiting 
d951ca58269a: Waiting 
8b29394cbf65: Waiting 
ae6c14b47b5b: Waiting 
cc3d128bf64a: Waiting 
0ffd9da7ab09: Waiting 
5f70bf18a086: Preparing 
50e8a6a6b86d: Waiting 
86887f413a79: Waiting 
1ce38f4e1b95: Waiting 
08e1eb39b3d4: Waiting 
3b4c66d4763e: Waiting 
ef4e5caf07dd: Waiting 
8dbde2919dc0: Waiting 
6ba17642a337: Waiting 
cc5d35837d4e: Waiting 
83f51a6277e1: Waiting 
26f6364d87a4: Waiting 
8ef42130d581: Waiting 
01bdd2f5dafd: Waiting 
ff8a042b4a8f: Waiting 
efc3c236c19f: Waiting 
0487cefc9dd5: Waiting 
6d8f301dd0cb: Waiting 
02e126437d86: Waiting 
c2e5344e77ca: Waiting 
50c88dcec6f4: Waiting 
c9d9c0a1ed90: Waiting 
2a8ecfb45c4b: Waiting 
24e285bc6c4b: Waiting 
48d342402e4f: Waiting 
cdcbf0fc3588: Waiting 
5f81249b0832: Waiting 
eb65aaac71f9: Waiting 
361d25cf52f1: Waiting 
214e78aa7550: Waiting 
107d42a8a9cd: Waiting 
e3d770e0d222: Waiting 
ebcd7c8c1491: Waiting 
3958e3868248: Waiting 
dcd8f84b6cdf: Waiting 
03459842c374: Waiting 
4624d7c4ac11: Waiting 
8367ba9317ed: Waiting 
6e2b713af791: Waiting 
5f5e8260b3f8: Waiting 
8e478c16644d: Waiting 
63ea072d5704: Waiting 
fff9055bdf2e: Waiting 
0b35c5c91607: Waiting 
68a85fa9d77e: Waiting 
denied: Your authorization token has expired. Reauthenticate and try again.

in the middle of a push... and its automated via a script, so there isn't a second in between...

I reinstalled v1 from v1/stable as you suggested.

$ snap list 
Name               Version          Rev    Tracking       Publisher   Notes
aws-cli            1.31.1           362    v1/stable      aws✓        classic

it is indeed newer. but broken on my system...

python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
python3.8: /snap/core20/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
unsupported version of aws-cli

reverting to latest/beta hopefully I get the working 1.27.51 argh,... I should not have touch this...

[mathieujobin]

its unfortunate, I can only revert to

$ snap install  aws-cli --channel latest/stable --classic
Fetch and check assertions for snap "aws-cli" (130)                                                                                             \
aws-cli 1.15.58 from Amazon Web Services (aws✓) installed

but it is also broken..

/snap/aws-cli/130/usr/bin/python3: /snap/core/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
/snap/aws-cli/130/usr/bin/python3: /snap/core/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.26' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
/snap/aws-cli/130/usr/bin/python3: /snap/core/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
...

1.27.51 was working perfectly.

[mathieujobin]

so my system is now broken, I can't do docker push anymore... what's left for me to do ? install via python tooling ?

[mathieujobin]

I am saved !!!

$ snap install aws-cli --revision 176 --classic
Warning: flag --classic ignored for strictly confined snap aws-cli

aws-cli 1.27.51 from Amazon Web Services (aws✓) installed

this is the working version.

[marrek-az]

I am saved !!!

$ snap install aws-cli --revision 176 --classic
Warning: flag --classic ignored for strictly confined snap aws-cli

aws-cli 1.27.51 from Amazon Web Services (aws✓) installed

this is the working version.

The version in the latest/stable channel hasn't been updated in awhile. To stay on the latest revision, I recommend moving from latest/stable to v1/stable if you need to use v1. snap refresh aws-cli --channel=v1/stable --classic

As for the issue with v2, It doesn't look like there is enough information in your last comment to reproduce, or even determine if it is a snap issue. I would suggest opening a separate issue and providing as much information as possible. If it's snap-related, I will try to determine what is going on.

[marrek-az]

@mathieujobin

$ snap install aws-cli --revision 176 --classic
Warning: flag --classic ignored for strictly confined snap aws-cli

One more thing I forgot to mention: installing by revision number could be risky, and you should use this with extreme caution. If a package was uploaded to v1/beta and then discarded due to problems, it will still show up as a numbered revision. That metadata is not available for you to query. I believe in the 100's we did some experimentation with strict confinement and determined that it caused a lot of problems we weren't prepared to address, so we switched back to classic confinement. Revision 176 is quite possibly broken.

The best recommendation is to install from v1/stable and use that version.

[mathieujobin]

@marrek-az as shown above, the version in v1/stable is broken.

rev 176 is the only version that works for me.

[marrek-az]

@mathieujobin I apologize for misunderstanding.

Installing from latest/beta wouldn't do any good at the moment, as that is currently tracking latest/candidate, which is 2.14.3. There is a reason why I don't recommend using "latest". Revision 176 was an abandoned beta which shouldn't be used for production long-term. I'm glad it's working for you for the moment.

I tested with 1.31.1 and with 1.31.2 in v1/candidate, and with 2.13.38, both using

aws ecr get-login-password

and it worked fine for me in all cases. I cannot reproduce the issue you are seeing. These tests were performed on an Ubuntu 22.04 host. I will attempt on other Ubuntu releases as soon as I can to see if I can reproduce the error.

The current v1 snap contains its own Python 3.8 interpreter, as the aws-cli's Python requirements were no longer compatible with the system Python shipped with Ubuntu 18.04 (which is supported for another 5 years under Ubuntu Pro). Embedding the interpreter was a workaround to support v1 on that release for as long as Ubuntu Pro and aws-cli v1 are both supported. It's possible that you are experiencing side-effects from that, but I am not currently able to identify the cause.