cnstlungu / portable-data-stack-dagster

A portable Datamart and Business Intelligence suite built with Docker, Dagster, dbt, DuckDB, PostgreSQL and Superset
MIT License
181 stars 37 forks source link

docker compose up --build fails #15

Closed eracle closed 2 months ago

eracle commented 2 months ago
dagster._core.errors.DagsterUserCodeLoadError: Error occurred during the loading of Dagster definitions in
executable_path=/usr/local/bin/python3, working_directory=/dagster-dbt/postcard_company_dm, package_name=postcard_company_dm
  File "/usr/local/lib/python3.11/site-packages/dagster/_grpc/server.py", line 411, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
                                                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster/_grpc/server.py", line 234, in __init__
    with user_code_error_boundary(
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/dagster/_core/errors.py", line 297, in user_code_error_boundary
    raise new_error from e
The above exception was caused by the following exception:
pydantic.errors.PydanticUserError: `TableMetadataSet` is not fully defined; you should define `TableColumn`, then call `TableMetadataSet.model_rebuild()`.

For further information visit https://errors.pydantic.dev/2.9/u/class-not-fully-defined
  File "/usr/local/lib/python3.11/site-packages/dagster/_core/errors.py", line 287, in user_code_error_boundary
    yield
  File "/usr/local/lib/python3.11/site-packages/dagster/_grpc/server.py", line 245, in __init__
    loadable_targets = get_loadable_targets(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster/_grpc/utils.py", line 60, in get_loadable_targets
    else loadable_targets_from_python_package(package_name, working_directory)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster/_core/workspace/autodiscovery.py", line 44, in loadable_targets_from_python_package
    module = load_python_module(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster/_core/code_pointer.py", line 134, in load_python_module
    return importlib.import_module(module_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/dagster-dbt/postcard_company_dm/postcard_company_dm/__init__.py", line 29, in <module>
    defs = Definitions(assets=load_assets_from_dbt_project(DBT_PROJECT_PATH, profiles_dir=DBT_PROFILES, use_build_command=True), resources=model_resources, jobs=[run_dbt_test_job])
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster/_core/decorator_utils.py", line 203, in wrapped_with_pre_call_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_defs.py", line 611, in load_assets_from_dbt_project
    return _load_assets_from_dbt_manifest(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_defs.py", line 967, in _load_assets_from_dbt_manifest
    dbt_assets_def = _dbt_nodes_to_assets(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_defs.py", line 407, in _dbt_nodes_to_assets
    ) = get_asset_deps(
        ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_utils.py", line 758, in get_asset_deps
    dagster_dbt_translator.get_metadata(dbt_resource_props),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_defs.py", line 940, in get_metadata
    return node_info_to_definition_metadata_fn(dbt_resource_props)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_utils.py", line 428, in default_metadata_from_dbt_resource_props
    **TableMetadataSet(
      ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dagster/_core/definitions/metadata/metadata_set.py", line 125, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/pydantic/main.py", line 211, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_mock_val_ser.py", line 99, in __getattr__
    raise PydanticUserError(self._error_message, code=self._code)
cnstlungu commented 2 months ago

Hi @eracle!

Thanks a lot for raising this issue.

I've just pushed a fix.

Reason: https://github.com/dagster-io/dagster/issues/24288