dagster-dbt : unable to load dbt project of an dbt-teradata adapter (when database is None in manifest.json)

[tallamohan]

Dagster version

1.7.13

What's the issue?

In dbt-teradata adapter, the database value is set as None. It considers only the schema value But in the latest dagster-dbt, asset_utils.py tries to build relation_name using the "database" value in the manifest.json file.

I am trying to add a sample dbt project as an asset using this tutorial: https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/load-dbt-models

But after staring the "dagster dev" server, I am seeing this error:

dagster._core.errors.DagsterUserCodeLoadError: Error occurred during the loading of Dagster definitions in
executable_path=C:\Users\mt255026\dbt-dagster-teradata\venv_py\Scripts\python.exe, module_name=jaffle_dagster.definitions, working_directory=C:\Users\mt255026\dbt-dagster-teradata\jaffle_shop\jaffle_dagster
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster\_grpc\server.py", line 411, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster\_grpc\server.py", line 234, in __init__
    with user_code_error_boundary(
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster\_core\errors.py", line 297, in user_code_error_boundary
    raise new_error from e
The above exception was caused by the following exception:
TypeError: sequence item 0: expected str instance, NoneType found
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster\_core\errors.py", line 287, in user_code_error_boundary
    yield
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster\_grpc\server.py", line 245, in __init__
    loadable_targets = get_loadable_targets(
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster\_grpc\utils.py", line 50, in get_loadable_targets
    else loadable_targets_from_python_module(module_name, working_directory)
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster\_core\workspace\autodiscovery.py", line 31, in loadable_targets_from_python_module
    module = load_python_module(
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster\_core\code_pointer.py", line 134, in load_python_module
    return importlib.import_module(module_name)
  File "C:\Users\mt255026\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\mt255026\dbt-dagster-teradata\jaffle_shop\jaffle_dagster\jaffle_dagster\definitions.py", line 6, in <module>
    from .assets import jaffle_shop_dbt_assets
  File "C:\Users\mt255026\dbt-dagster-teradata\jaffle_shop\jaffle_dagster\jaffle_dagster\assets.py", line 7, in <module>
    @dbt_assets(manifest=dbt_manifest_path)
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster\_core\decorator_utils.py", line 223, in wrapped_with_context_manager_fn
    return fn(*args, **kwargs)
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster_dbt\asset_decorator.py", line 311, in dbt_assets
    ) = build_dbt_multi_asset_args(
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster_dbt\asset_utils.py", line 849, in build_dbt_multi_asset_args
    **dagster_dbt_translator.get_metadata(dbt_resource_props),
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster_dbt\dagster_dbt_translator.py", line 220, in get_metadata
    return default_metadata_from_dbt_resource_props(dbt_resource_props)
  File "C:\Users\mt255026\dbt-dagster-teradata\venv_py\lib\site-packages\dagster_dbt\asset_utils.py", line 464, in default_metadata_from_dbt_resource_props
    relation_name = ".".join(

What did you expect to happen?

Successfully load dbt project of dbt-teradata adapter

How to reproduce?

Load any dbt project which uses dbt-teradata adapter

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[amaralvieira]

This macro is the one generating the None database:

{% macro teradata__generate_database_name(custom_database_name=none, node=none) -%}
  {% do return(None) %}
{%- endmacro %}

I've found the same logic in a few other dbt adapters, namely:

• dbt-starrocks
• dbt-impala
• dbt-hive
• dbt-spark
• dbt-mysql , as reported by murs006 below

It would be great to have dagster just skip over the database key in these cases.

[tallamohan]

Yes @amaralvieira , same as dbt-teradata a few adapters do not consider database.

[amaralvieira]

This is the code in dagster-dbt which is throwing the error:

https://github.com/dagster-io/dagster/blob/36968cc4598b7e5a9fedb4f26a93fbde93675cd0/python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py#L459-L470

dbt_resource_props["database"] should be skipped when None to accommodate the adapters above.

[murs006]

I encountered a similar issue while setting up a project with dbt and MySQL. The setup works perfectly when I use PostgreSQL in the dbt profiles, but it fails consistently with MySQL. As mentioned previously, the dbt-mysql adapter is the issue I believe.

[dargmuesli]

I've submitted a PR: https://github.com/dagster-io/dagster/pull/24021 Please give it a review if you know dagster's code base a bit :wink: