[CT-1185] [Feature] [Spike] Support dbt Python models on Redshift/AWS - Githubissues

dbt-labs / dbt-redshift

dbt-redshift contains all of the code enabling dbt to work with Amazon Redshift

https://getdbt.com

Apache License 2.0

98 stars 59 forks source link

[CT-1185] [Feature] [Spike] Support dbt Python models on Redshift/AWS #188

Closed lostmygithubaccount closed 1 year ago

lostmygithubaccount commented 2 years ago

UPDATE: closed in favor of https://github.com/dbt-labs/dbt-redshift/issues/204

Is this your first time submitting a feature request?

[X] I have read the expectations for open source contributors
[X] I have searched the existing issues, and I could not find an existing issue for this feature
[X] I am requesting a straightforward extension of existing dbt-redshift functionality, rather than a Big Idea better suited to a discussion

Describe the feature

dbt Python models, but on Redshift/AWS

Describe alternatives you've considered

n/a

Who will this benefit?

Redshift/AWS users

Are you interested in contributing this feature?

not exactly...

Anything else?

seems like the Python cursor API (thanks @dataders) supports reading and writing dataframes: https://docs.aws.amazon.com/redshift/latest/mgmt/python-api-reference.html#python-api-cursor

potentially combine that with something like this: https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/launch-a-spark-job-in-a-transient-emr-cluster-using-a-lambda-function.html

we should investigate the technical feasibility here

dbeatty10 commented 1 year ago

Closing as a duplicate of https://github.com/dbt-labs/dbt-redshift/issues/204.

This actually pre-dates #204 by about 4 weeks, but the latter has implementation details and some discussion.