aws / aws-sdk-pandas

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
https://aws-sdk-pandas.readthedocs.io
Apache License 2.0
3.94k stars 702 forks source link

A public json to identify the right layer ARN #2434

Closed madhavvishnubhatta closed 1 year ago

madhavvishnubhatta commented 1 year ago

Is your feature request related to a problem? Please describe. At the moment, there is no good way to identify what the lambda layer ARN is for a given region, python version and package version. The layer ARNs need to be hardcoded, and it becomes tricky if we have to deploy lambda functions that use this package to multiple regions.

Describe the solution you'd like A public json that maps every combination of region, python version and package version to a specific layer ARN.

Describe alternatives you've considered While the json solves the problem for this package, publishing the ARNs as public parameters on SSM parameter store might be a better way as that would apply to all AWS vended/curated layers (not just this one). But that might be a longer term approach, while this json could help with the short term.

Additional context Based on this Slack discussion.

P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.

jaidisido commented 1 year ago

Thanks @madhavvishnubhatta, we have a preference for the public SSM solution. Someone in the team will work on it

malachi-constant commented 1 year ago

Public parameters are now in place and can be found in both the SSM console under Public Parameters | aws-sdk-pandas

Screenshot 2023-10-09 at 8 16 04 AM

Or via the CLI

aws ssm get-parameters-by-path --path /aws/service/aws-sdk-pandas --recursive

Will add to our docs shortly.

madhavvishnubhatta commented 1 year ago

Looks like these public parameters are available only in us-east-1 but not in other regions. Is that the case? If so, is that intentional?

malachi-constant commented 1 year ago

@madhavvishnubhatta Thanks for raising this. We agree that global parameters should be stored in the region where the layer exists. We will reopen and update this ticket with progress.

malachi-constant commented 1 year ago

SSM parameters are now up to date <=3.4.2 in all regions (including opt-in). Note: Parameters reside in the same region as their layer now.