fal-ai / dbt-fal

do more with dbt. dbt-fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.
https://fal.ai/dbt-fal
Apache License 2.0
853 stars 72 forks source link

fal flow run --target TARGET loads non-flag target before switching #744

Open chamini2 opened 1 year ago

chamini2 commented 1 year ago

Describe the bug When running fal flow run --target ddb (my duckdb profile) I get the following error:

20:46:15  Error importing adapter: No module named 'dbt.adapters.fal'
Traceback (most recent call last):
  File "/Users/matteo/.pyenv/versions/3.8.13/envs/dbtdd/lib/python3.8/site-packages/dbt/adapters/factory.py", line 56, in load_plugin
    mod: Any = import_module("." + name, "dbt.adapters")
  File "/Users/matteo/.pyenv/versions/3.8.13/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dbt.adapters.fal'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/matteo/.pyenv/versions/3.8.13/envs/dbtdd/lib/python3.8/site-packages/dbt/config/profile.py", line 179, in _credentials_from_profile
    cls = load_plugin(typename)
  File "/Users/matteo/.pyenv/versions/3.8.13/envs/dbtdd/lib/python3.8/site-packages/dbt/adapters/factory.py", line 201, in load_plugin
    return FACTORY.load_plugin(name)
  File "/Users/matteo/.pyenv/versions/3.8.13/envs/dbtdd/lib/python3.8/site-packages/dbt/adapters/factory.py", line 62, in load_plugin
    raise RuntimeException(f"Could not find adapter type {name}!")
dbt.exceptions.RuntimeException: Runtime Error
  Could not find adapter type fal!

And the ddb target has type: duckdb

fal_test:
  target: fal_ddb

  outputs:
# relevant section of my profiles.yml
    fal_ddb:
      type: fal
      db_profile: ddb
    ddb:
      type: duckdb
      path: "/Users/user/duck_db_dbt_dump.db" 
      s3_region: "us-east"
      s3_access_key_id: aa
      s3_secret_access_key: "xx"

Your environment

❯ dbt --version Core:

Plugins:

How to reproduce Point to a not-installed profile in your target: property in profiles.yml, and pass a yes-installed profile.

Expected behavior Should completely ignore the target: of profiles.yml and instead use --target setting.

Actual behavior It will load the profile with the not-installed profile before actually using the correct one, and raise an exception.

iddar commented 1 year ago

Same issue here on mac os user brew version.

❯ dbt --version
Core:
  - installed: 1.5.0
  - latest:    1.5.1 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - postgres: 1.5.0 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

❯ fal --version
fal 0.9.1

error:

❯ dbt run --target fal_dev
23:36:05  Running with dbt=1.5.0
23:36:05  Error importing adapter: No module named 'dbt.adapters.fal'
23:36:05  Encountered an error:
Runtime Error
  Credentials in profile "flux_analytics", target "fal_dev" invalid: Runtime Error
    Could not find adapter type fal!
chamini2 commented 1 year ago

Hello, @iddar

From your output it seems you do not have dbt-fal installed. fal and dbt-fal are two separate packages. To use dbt with a fal type adapter, you need to install dbt-fal.

Can you run pip install dbt-fal==1.5.2 and let us know if know it works?