AllenNeuralDynamics / aind-data-access-api

Library to interface with AIND databases

MIT License

2 stars 0 forks source link

Use redshift_connector #41

Open mekhlakapoor opened 3 months ago

mekhlakapoor commented 3 months ago

User story

Currently, pandas is pinned to be between 2.0. and 2.2 which limits users. As a developer, I want to replace sqlalchemy and psycopg2-binary dependencies with redshift_connector so that we don't need to worry about compatibility of pandas with other dependencies

Acceptance criteria

[ ] Given staging bucket, copy table to bucket
[ ] Given table in staging bucket, copy to rds to overwrite table

Sprint Ready Checklist

[ ] 1. Acceptance criteria defined
[ ] 2. Team understands acceptance criteria
[ ] 3. Team has defined solution / steps to satisfy acceptance criteria
[ ] 4. Acceptance criteria is verifiable / testable
[ ] 5. External / 3rd Party dependencies identified
[ ] 6. Ticket is prioritized and sized

Notes

Redshift Connector uses S3 as a middle ground. We'll need to create a temp bucket https://stackoverflow.com/questions/76532906/unrecognized-configuration-parameter-standard-conforming-strings-while-queryin https://docs.aws.amazon.com/redshift/latest/mgmt/python-connect-examples.html

Add any helpful notes here.

mekhlakapoor commented 2 months ago

Might be useful package: https://pypi.org/project/awswrangler/