Open stefannica opened 10 months ago
Thanks for the report @stefannica. One detail to confirm: what's the output of AWS_PROFILE=skypilot aws configure list
?
Relevant code:
I used skypilot
as an example, but the exact contents of the AWS profile can be an IAM role or even an STS token:
$ AWS_PROFILE=zenml-7f225511 aws configure list
Name Value Type Location
---- ----- ---- --------
profile zenml-7f225511 env ['AWS_DEFAULT_PROFILE', 'AWS_PROFILE']
access_key ****************7U7F shared-credentials-file
secret_key ****************FQ7G shared-credentials-file
region eu-central-1 config-file ~/.aws/config
Name Value Type Location
---- ----- ---- --------
profile connectors env ['AWS_DEFAULT_PROFILE', 'AWS_PROFILE']
access_key ****************BG6D assume-role
secret_key ****************GxVD assume-role
region us-east-1 config-file ~/.aws/config
By the way, we solved a similar problem in ZenML: picking up AWS credentials from the client machine and storing them somewhere else. The relevant ZenML code doesn't explicitly look at the AWS configuration files and doesn't even use the AWS CLI Python package, just plain boto3 and botocore, which makes it easier to detach from the AWS setup and probably more portable. I'll reference this here, if you think it may help: https://github.com/zenml-io/zenml/blob/main/src/zenml/integrations/aws/service_connectors/aws_service_connector.py#L1389-L1643
I used
skypilot
as an example, but the exact contents of the AWS profile can be an IAM role or even an STS token:
- STS token credentials:
$ AWS_PROFILE=zenml-7f225511 aws configure list Name Value Type Location ---- ----- ---- -------- profile zenml-7f225511 env ['AWS_DEFAULT_PROFILE', 'AWS_PROFILE'] access_key ****************7U7F shared-credentials-file secret_key ****************FQ7G shared-credentials-file region eu-central-1 config-file ~/.aws/config
- IAM role credentials:
Name Value Type Location ---- ----- ---- -------- profile connectors env ['AWS_DEFAULT_PROFILE', 'AWS_PROFILE'] access_key ****************BG6D assume-role secret_key ****************GxVD assume-role region us-east-1 config-file ~/.aws/config
To confirm: Which of these 2 cases wasn't working in the original issue description?
It seems like the first case (STS -> shared-credentials-file) should be handled by our current code? If not, something needs to be changed. (E.g., maybe the temporary credentials were auto-stored at a different path than ~/.aws/credentials
.)
For the second case, assume-role
, it seems like we simply need to add an enum like this and mimic its handling:
In either case, would love your help in patching this!
To confirm: Which of these 2 cases wasn't working in the original issue description?
@concretevitamin neither of them were working, because they both use the AWS_PROFILE
environment variable and, as I understand it, currently this is not supported.
It seems like the first case (STS -> shared-credentials-file) should be handled by our current code? If not, something needs to be changed. (E.g., maybe the temporary credentials were auto-stored at a different path than
~/.aws/credentials
.)
The credentials were indeed stored in ~/.aws/credentials
, but not under the default AWS profile.
For the second case,
assume-role
, it seems like we simply need to add an enum like this and mimic its handling:In either case, would love your help in patching this!
In either case, would love your help in patching this!
Sure, I'd love to help out with this. I'll try to set out some time for it over the next few days.
This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.
I had the same issue, for anyone facing the issue, as credentials file is copied, use sky --env AWS_PROFILE=aws_profile_to_use ...
this sets the profile to use on cluster as env variable, and you can assign the was profile with the right permissions
If the
AWS_PROFILE
environment variable is set on the client, this is not carried over on the AWS EC2 VM side and may cause problems executing subsequent tasks because of lack of permissions because the VM uses the default AWS configuration profile.How to reproduce:
skypilot
with credentials required by SkyPilot. To reproduce this problem properly, the default AWS configuration profile should not have these permissionsAWS_PROFILE
environment variableThis command should work without problems:
Looking at the ray cluster logs inside the VM, there should be some AWS permissions errors due to the fact that the VM is trying to use the default AWS configuration profile instead of the one passed to the client, e.g.: