fal-ai / dbt-fal

do more with dbt. dbt-fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.
https://fal.ai/dbt-fal
Apache License 2.0
851 stars 71 forks source link

issue with postgres source automagic using dbt 1.3.1 #735

Open dg-data opened 1 year ago

dg-data commented 1 year ago

Describe the bug Fal could not read dbt source.

Your environment

sources:

Actual behavior

dbt run --profiles-dir ./config
14:11:53  Found 2 models, 0 tests, 0 snapshots, 0 analyses, 289 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
14:11:54  Concurrency: 1 threads (target='pre')
14:11:55  Finished running  in 0 hours 0 minutes and 2.16 seconds (2.16s).

fal --debug --profiles-dir ./config run
15:01:52  Found 2 models, 0 tests, 0 snapshots, 0 analyses, 289 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
15:01:52  [WARNING]: Could not read dbt sources artifact

From python:

from fal import FalDbt
faldbt = FalDbt(profiles_dir="./dades-dag-sandbox/dbt_dades_sandbox/config", project_dir="./dades-dag-sandbox/dbt_dades_sandbox")
faldbt.list_sources()

output: [DbtSource(name='dades_sandbox', tests=[], status='skipped')]

$~$
df = faldbt.source('dades_sandbox', 'rnc_database')

File /opt/conda/lib/python3.10/site-packages/fal/telemetry/telemetry.py:349, in log_call.<locals>._log_call.<locals>.wrapper(*func_args, **func_kwargs)
    346 start = datetime.datetime.now()
    348 try:
--> 349     result = func(*func_args, **func_kwargs)
    350 except Exception as e:
    351     log_api(
    352         action=f"{action}_error",
    353         total_runtime=str(datetime.datetime.now() - start),
   (...)
    358         },
    359     )

File /opt/conda/lib/python3.10/site-packages/faldbt/project.py:682, in FalDbt.source(self, target_source_name, target_table_name)
    676 """
    677 Download a dbt source as a pandas.DataFrame automagically.
    678 """
    680 target_source = self._source(target_source_name, target_table_name)
--> 682 return lib.fetch_target(
    683     self.project_dir,
    684     self.profiles_dir,
    685     target_source,
    686     self._profile_target,
    687     config=self._config,
    688 )

File /opt/conda/lib/python3.10/site-packages/faldbt/lib.py:236, in fetch_target(project_dir, profiles_dir, target, profile_target, config, adapter)
    233 relation = _get_target_relation(adapter, target)
    235 if relation is None:
--> 236     raise Exception(f"Could not get relation for '{target.unique_id}'")
    238 return _fetch_relation(adapter, relation)

Exception: Could not get relation for 'source.dbt_dades_sandbox.dades_sandbox.rnc_database'

Additional context Postgres user reader has read-only access.