How to use with AWS CLI v2

ches commented 4 years ago

I'll send a PR for the README if I have a few more minutes, but thought I'd share how to get the plugin working in the v2 CLI for others that may see it here—it took finding separate answers for a couple of issues:

Use the CLI's bundled Python with pip install --no-deps awscli-plugin-endpoint. (This will mean something like /usr/local/opt/awscli/libexec/bin/pip depending on installation).

Add the new cli_legacy_plugin_path setting in ~/.aws/config—see aws/aws-cli#4854:

[plugins]
cli_legacy_plugin_path = /usr/local/opt/awscli/libexec/lib/python3.8/site-packages
endpoint = awscli_plugin_endpoint

They don't want to release v2 on PyPI. So without --no-deps I was getting:

$ /usr/local/opt/awscli/libexec/bin/pip install awscli-plugin-endpoint
[snip]
Requirement already satisfied: awscli>=1.11.0 in /usr/local/Cellar/awscli/2.0.0_1/libexec/lib/python3.8/site-packages (from awscli-plugin-endpoint) (2.0.0)
ERROR: No matching distribution found for botocore==2.0.0dev4 (from awscli>=1.11.0->awscli-plugin-endpoint)

It looks like the plugin's awscli>=1.11.0 dependency triggers pip to try to resolve deps including the CLI's, and fails because botocore v2 releases aren't published to PyPI either.

After these two steps, the plugin is working for me. Thanks for making it!

$ aws --version
aws-cli/2.0.0 Python/3.8.2 Darwin/18.7.0 botocore/2.0.0dev7

wbingli commented 4 years ago

Thanks a lot for the info. Will update to the README.

rbrianhazzard commented 3 years ago

Hi @wbingli - the readme still says pip install awscli-plugin-endpoint regardless of cli version. This only worked for the cli v2 when I did pip install --no-deps awscli-plugin-endpoint. Could you please call that out in the readme?

Thanks for the awesome plugin!

johnlabarge commented 3 years ago

This still isn't working for me. How did you get alwscli to load the modules? I always get e.g. ModuleNotFoundError: No module named '/usr/local/opt/awscli/libexec/lib/python3/site-packages' At first I thought it was the dot in python3.9 so I removed it but there seems to be something wrong with the import.

ches commented 3 years ago

@johnlabarge A bit late, but in case it helps anyone else… check which aws, view the file and make certain the shebang is pointing at awscli's bundled Python installation. You might also notice the wrong Python in the other lines in the backtrace of the ModuleNotFoundError.

Your problem might arise because the aws executable got rewritten, an easy mistake to make by doing pip install awscli-plugin-endpoint with the pip on your path instead of awscli's embedded one, and by leaving out --no-deps which will result in awscli (1.x) getting installed by pip, as a dependency…

I know the ModuleNotFoundError line refers to /usr/local/opt/awscli, so it leads one to think aws is using the right Python, but the error stems from cli_legacy_plugin_path.

Probable solution if it's a Homebrew setup:

$ pip3 uninstall awscli
$ brew reinstall awscli
$ /usr/local/opt/awscli/libexec/bin/pip install --no-deps awscli-plugin-endpoint

The need to redo this delicate dance correctly every time awscli gets a version update is about to drive me to doing it with a Dockerfile and using the image, instead of aws from Homebrew installation…

clayg commented 2 months ago

what the heck is going on with awscliv2 - it's a python module that runs a docker container? My aws command is working fine w/ the endpoint-plugin. The entrypoint for the v2 runtime is awsv2 - totally different name on the shell $PATH.

Won't the plugin have to be installed IN the container - maybe there's some bind mounts going on? But then wouldn't that mean the plugin path should just be the same site-packages/dist-packages path that awscliv2 is installed into?

https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration-changes.html#cliv2-migration-profile-plugins

clayg commented 2 months ago

so, I'm going to set this down... but for the next person to pick up...

vagrant@saio:~$ awsv2 --version
2.3.0
AWS CLI v2 command: docker run -i --rm -v /home/vagrant/.aws:/root/.aws -v /home/vagrant:/aws amazon/aws-cli

No module named 'awscli_plugin_endpoint'

getting in to the container with my system site-packages mounted in helped a lot

$ docker run -it --entrypoint /bin/bash -v /home/vagrant/.aws:/root/.aws -v /home/vagrant:/aws -v /home/vagrant/.local/lib/python3.10/site-packages:/plugins amazon/aws-cli

with a ~/.aws/config that had a [plugins] section like

[plugins]
cli_legacy_plugin_path = /plugins
endpoint = awscli_plugin_endpoint

Then at least when I run the real entrypoint from the image it was able to load the plugin:

bash-4.2# /usr/local/bin/aws s3api list-buckets

Could not connect to the endpoint URL: "http://saio:8080/"

wbingli / awscli-plugin-endpoint

How to use with AWS CLI v2 #15