lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

Why do AWS credentials have to be hard coded into config files? #1293

Closed rabernat closed 6 months ago

rabernat commented 6 months ago

Thanks for your work on this great library! For context, I collaborate with @tomwhite on cubed which targets Lithops as an execution engine.

I am trying to get Lithops running on AWS lambda, and I'm puzzled by the requirement that AWS access key and secret have to be hard coded into a config file. This feels counter to best practices for AWS authentication. In general, it would be nice to have the same authentication precedence options as in the AWS CLI.

In particular, I would like to

Is something like this on the roadmap?

rabernat commented 6 months ago

Following up on this a bit...

It looks like it would be sufficient to just leave our / set to None aws_access_key_id and aws_secret_access_key in these lines. boto3 will automatically detect credentials in the correct way.

https://github.com/lithops-cloud/lithops/blob/b199505955f11602331e4725bf1844f885e2f7f8/lithops/serverless/backends/aws_lambda/aws_lambda.py#L66-L70

So simply removing this check should fix my issue:

https://github.com/lithops-cloud/lithops/blob/b199505955f11602331e4725bf1844f885e2f7f8/lithops/serverless/backends/aws_lambda/config.py#L73-L74

JosepSampe commented 6 months ago

Make sense, there was a WIP in https://github.com/lithops-cloud/lithops/pull/1114 but it wasn't finished. I will check it

JosepSampe commented 6 months ago

@rabernat I add this patch that includes the required changes for this, and a necessary step to allow this. Feel free to try it out and provide feedback

rabernat commented 6 months ago

It works!