fal-ai / dbt-fal

do more with dbt. dbt-fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.
https://fal.ai/dbt-fal
Apache License 2.0
853 stars 72 forks source link

chore: merge dbt-fal and fal packages into dbt-fal #882

Closed chamini2 closed 11 months ago

chamini2 commented 11 months ago

The idea of this PR is to merge the 2 packages into one, and we would start releasing only the dbt-fal package.

New dbt-fal command line tool

We now install the fal and dbt-fal CLIs, they are the same but the fal one gets a warning when started.

❯ fal flow run
The dbt tool `fal` and `dbt-fal` adapter have been merged into a single tool.
Please use the new `dbt-fal` command line tool instead.
Running `pip install dbt-fal` will install the new tool and the adapter alongside.
Then run your command like

    dbt-fal <command>

00:29:04  Running with dbt=1.5.1
00:29:04  Unable to do partial parsing because config vars, config profile, or config target have changed
...

❯ dbt-fal flow run
00:29:10  Running with dbt=1.5.1
00:29:10  Unable to do partial parsing because config vars, config profile, or config target have changed
...

Warning on fal imports

Another thing to consider is that now imports from fal moved to fal.dbt. Now we get warnings for fal direct imports.

>>> import fal
>>> fal.FalDbt
The dbt tool `fal` and `dbt-fal` adapter have been merged into a single tool.
Please import from the `fal.dbt` module instead.
Running `pip install dbt-fal` will install the new tool and the adapter alongside.
Then import from the `fal.dbt` module like

    from fal.dbt import FalDbt

<class 'fal.dbt.utils.FalDbt'>
>>> fal.FalDbt # no more warnings
<class 'fal.dbt.utils.FalDbt'>
>>> from fal import NodeStatus
The dbt tool `fal` and `dbt-fal` adapter have been merged into a single tool.
Please import from the `fal.dbt` module instead.
Running `pip install dbt-fal` will install the new tool and the adapter alongside.
Then import from the `fal.dbt` module like

    from fal.dbt import NodeStatus

>>> from fal import NodeStatus # no more warnings
>>> from fal.dbt import Context

To later free the fal namespace.


Run tests for:


Closes FEA-1227 Closes FEA-1263 Closes FEA-1267

linear[bot] commented 11 months ago

FEA-1227 merge dbt-fal and fal packages into dbt-fal PyPI package (careful with version)

chamini2 commented 11 months ago

Will a person be able to use fal-serverless (new fal) and dbt-fal? I see that we're putting the fal cli in fal namespace, so just curious if the CLI and serverless functions can work together in the same environment.

Even though fal cli is in the fal namespace, notice that the whole project is in src/fal/dbt/*. That means that to import anything from it you use import fal.dbt.

This is the same behavior we see for dbt adapters. They install modules in the dbt namespace, but inside the dbt.adapters.<adapter-name> module, so it does not conflict with imports from dbt-core .

Let me know if I did not explain myself well and I can clarify!

chamini2 commented 11 months ago

Some tests are set to @todo, does this mean that this feature doesn't work for those databases for both CLI and adapter?

The tests right now keep being separate, we test the cli and adapter separately.

So a todo in the cli_tests folder applies to cli and a todo in the integration_tests folder applies to adapter.

linear[bot] commented 11 months ago

FEA-1263 Change dbt-fal / fal-CLI integration tests to run weekly

FEA-1267 Add warning about fal cli and imports going away soon

burkaygur commented 11 months ago

🚢 thank you @chamini2