search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.94k stars 1.49k forks source link

Dagster DBT Error with Pydantic 2.9.0 #24264

Closed cody-scott closed 2 months ago

cody-scott commented 2 months ago

Dagster version

1.7.15

What's the issue?

After the recent pydantic update, dagster is throwing errors related to the Dagster DBT package

analytics_tests/validator_tests/test_schema_validator_integrations.py:4: in <module>
    from analytics.utils import get_target_db_connection_dict
analytics/__init__.py:7: in <module>
    from .assets import all_assets, all_asset_defs
analytics/assets/__init__.py:3: in <module>
    from . import (
analytics/assets/dbt/__init__.py:7: in <module>
    @dbt_assets(manifest=dbt_manifest_path)
/usr/local/lib/python3.11/site-packages/dagster/_core/decorator_utils.py:223: in wrapped_with_context_manager_fn
    return fn(*args, **kwargs)
/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_decorator.py:311: in dbt_assets
    ) = build_dbt_multi_asset_args(
/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_utils.py:849: in build_dbt_multi_asset_args
    **dagster_dbt_translator.get_metadata(dbt_resource_props),
/usr/local/lib/python3.11/site-packages/dagster_dbt/dagster_dbt_translator.py:220: in get_metadata
    return default_metadata_from_dbt_resource_props(dbt_resource_props)
/usr/local/lib/python3.11/site-packages/dagster_dbt/asset_utils.py:473: in default_metadata_from_dbt_resource_props
    **TableMetadataSet(
/usr/local/lib/python3.11/site-packages/dagster/_core/definitions/metadata/metadata_set.py:125: in __init__
    super().__init__(*args, **kwargs)
E   pydantic.errors.PydanticUserError: `TableMetadataSet` is not fully defined; you should define `TableColumn`, then call `TableMetadataSet.model_rebuild()`.
E   
E   For further information visit https://errors.pydantic.dev/2.9/u/class-not-fully-defined

What did you expect to happen?

Dagster to start successfully

How to reproduce?

install pydantic=2.9.0 with the dagster 1.7.15. An unpinned install should pull the latest 2.9.0 version

Deployment type

Docker Compose

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

garethbrickman commented 2 months ago

We suspect pydantic 2.9.0 rolling out a couple of hours ago as the root cause. Adding a pydantic<2.9.0 pin to your dependencies is the workaround. We will add this pin in the upcoming 1.8.5 release of Dagster.

cody-scott commented 2 months ago

Same issue with 1.8.4. Pinning to less then 2.9 was my fix too. Thanks!

cody-scott commented 2 months ago

thanks @garethbrickman appreciate the quick turn around!

AntonMaxen commented 2 months ago

Quick Work pinning to <2.9 made our instance boot again!

georgeteena commented 2 months ago

pydantic<2.9.0 worked for me as well.

jessanmen1 commented 2 months ago

Thanks for this issue, this was driving me crazy 😆