search

fal-ai / dbt-fal

do more with dbt. dbt-fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.
https://fal.ai/dbt-fal
Apache License 2.0
853 stars 74 forks source link

does write_to_source work cross database? #111

Open mike-seekwell opened 2 years ago

mike-seekwell commented 2 years ago

e.g. is it able to use postgres as the source and snowflake as the destination?

turbo1912 commented 2 years ago

Hey @mike-seekwell great question! This entirely depends on your dbt project. Fal right now is coupled to your dbt configuration. If you tell us a little bit about your usecase we can walk backwards from there and create a feature request.

It sounds like you are using fal as a standalone script and not at the end of a dbt run? or potentially you are using dbt to transform your postgres tables and want to write to snowflake at the end of a dbt run?

mike-seekwell commented 2 years ago

It's running as part of dbt

are using dbt to transform your postgres tables and want to write to snowflake at the end of a dbt run?

Yes

turbo1912 commented 2 years ago

Got it! We actually have spoken to several other people who want to connect to external targets.

I have created an issue for your feature request, https://github.com/fal-ai/fal/issues/115.

One open question we have is where to read the db credentials for the external source, we think dbt profiles.yml is the best place for it but would love to hear what you think?