search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
10.81k stars 1.35k forks source link

Overload of warning logs when using dbt-core 1.8.* #22763

Open ssillaots-boku opened 3 weeks ago

ssillaots-boku commented 3 weeks ago

Dagster version

1.7.11

What's the issue?

Dagster started supporting dbt-core 1.8. from 1.7.7. When updating dbt-core to 1.8. then there's an immense increase in ExperimentalWarnings and DeprecationWarnings logs. We are talking about a thousand rows of logs every minute compared to a couple before version bump. Examples:

2024-06-28 12:41:45 +0300 - dagster - WARNING - <project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `backfill_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
2024-06-28 12:41:45 +0300 - dagster - WARNING - <project path>/.venv/lib/python3.10/site-packages/dagster/_core/remote_representation/external_data.py:1782: DeprecationWarning: Support for arbitrary metadata values is deprecated and will be removed in 2.0.0. In the future, all user-supplied metadata values must be one of typing.Union[dagster._core.definitions.metadata.metadata_value.MetadataValue, dagster._core.definitions.metadata.table.TableSchema, dagster._core.definitions.asset_key.AssetKey, os.PathLike, typing.Dict[typing.Any, typing.Any], float, int, typing.List[typing.Any], str, datetime.datetime, NoneType]
2024-06-28 11:23:09 +0300 - dagster - WARNING - <project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_policy.py:328: ExperimentalWarning: Class `NotAssetCondition` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
  children.append(NotAssetCondition(operand=discard_condition))
2024-06-28 11:23:09 +0300 - dagster - WARNING - <project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_policy.py:331: ExperimentalWarning: Class `AndAssetCondition` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
  return AndAssetCondition(operands=children)
2024-06-28 11:31:07 +0300 - dagster - WARNING - <project path>/.venv/lib/python3.10/site-packages/dagster/_core/remote_representation/external_data.py:1782: DeprecationWarning: Support for arbitrary metadata values is deprecated and will be removed in 2.0.0. In the future, all user-supplied metadata values must be one of typing.Union[dagster._core.definitions.metadata.metadata_value.MetadataValue, dagster._core.definitions.metadata.table.TableSchema, dagster._core.definitions.asset_key.AssetKey, os.PathLike, typing.Dict[typing.Any, typing.Any], float, int, typing.List[typing.Any], str, datetime.datetime, NoneType]
2024-06-28 11:31:07 +0300 - dagster - WARNING - <project path>/.venv/lib/python3.10/site-packages/dagster/_core/remote_representation/external_data.py:481: DeprecationWarning: Property `ScheduleDefinition.environment_vars` is deprecated and will be removed in 2.0. Setting this property no longer has any effect.
2024-06-28 12:42:50 +0300 - dagster - WARNING - <project path>/.venv/lib/python3.10/site-packages/dagster/_core/remote_representation/external_data.py:1782: DeprecationWarning: Support for arbitrary metadata values is deprecated and will be removed in 2.0.0. In the future, all user-supplied metadata values must be one of typing.Union[dagster._core.definitions.metadata.metadata_value.MetadataValue, dagster._core.definitions.metadata.table.TableSchema, dagster._core.definitions.asset_key.AssetKey, os.PathLike, typing.Dict[typing.Any, typing.Any], float, int, typing.List[typing.Any], str, datetime.datetime, NoneType]
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_policy.py:213: ExperimentalWarning: Class `AutoMaterializePolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/backfill_policy.py:74: ExperimentalWarning: Class `BackfillPolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `auto_materialize_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `backfill_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_rule.py:92: DeprecationWarning: Class `MaterializeOnRequiredForFreshnessRule` is deprecated and will be removed in 1.8. Lazy auto-materialize is deprecated, in favor of explicit cron-based scheduling rules. Additional alternatives to replicate more of the lazy auto-materialize behavior will be provided before this is fully removed.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_policy.py:213: ExperimentalWarning: Class `AutoMaterializePolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/backfill_policy.py:62: ExperimentalWarning: Class `BackfillPolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `auto_materialize_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `backfill_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_rule.py:92: DeprecationWarning: Class `MaterializeOnRequiredForFreshnessRule` is deprecated and will be removed in 1.8. Lazy auto-materialize is deprecated, in favor of explicit cron-based scheduling rules. Additional alternatives to replicate more of the lazy auto-materialize behavior will be provided before this is fully removed.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_policy.py:213: ExperimentalWarning: Class `AutoMaterializePolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/backfill_policy.py:74: ExperimentalWarning: Class `BackfillPolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `auto_materialize_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `backfill_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_rule.py:92: DeprecationWarning: Class `MaterializeOnRequiredForFreshnessRule` is deprecated and will be removed in 1.8. Lazy auto-materialize is deprecated, in favor of explicit cron-based scheduling rules. Additional alternatives to replicate more of the lazy auto-materialize behavior will be provided before this is fully removed.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_policy.py:213: ExperimentalWarning: Class `AutoMaterializePolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/backfill_policy.py:74: ExperimentalWarning: Class `BackfillPolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `auto_materialize_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `backfill_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_rule.py:92: DeprecationWarning: Class `MaterializeOnRequiredForFreshnessRule` is deprecated and will be removed in 1.8. Lazy auto-materialize is deprecated, in favor of explicit cron-based scheduling rules. Additional alternatives to replicate more of the lazy auto-materialize behavior will be provided before this is fully removed.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/auto_materialize_policy.py:213: ExperimentalWarning: Class `AutoMaterializePolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/definitions/backfill_policy.py:74: ExperimentalWarning: Class `BackfillPolicy` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `auto_materialize_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
<project path>/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py:203: ExperimentalWarning: Parameter `backfill_policy` of function `asset` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.

I'm running project with poetry. Current versions are dbt-core 1.8.3, dbt-redshift 1.8.1. All dagster related versions are 1.7.11. But once again. The issue was the same when I used Dagster 1.7.7 and dbt-core/dbt-redshift was 1.8.0.

Right now I've added a warnings.simplefilter('ignore') in the code to keep log count lower. But even then some warnings still come through which is interesting in it's own right.

What did you expect to happen?

I expect Dagster to run without emitting loads of warnings.

How to reproduce?

With dbt-core 1.8.3, dbt-redshift 1.8.1 and Dagster 1.7.7+, poetry 1.6.1

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

NiallRees commented 3 weeks ago

Plus one. I had to revert because I was getting OOMs from the number of logs being emitted

NiallRees commented 3 weeks ago

Still getting some warnings when using dbt==1.7.*, but not as many:

/Users/niallwoodward/dev/select/web/backend/.venv/lib/python3.11/site-packages/dagster_dbt/dbt_project.py:193: ExperimentalWarning: Class `DagsterDbtManifestPreparer` is experimental. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at https://docs.python.org/3/library/warnings.html#describing-warning-filters.
  manifest_preparer: DbtManifestPreparer = DagsterDbtManifestPreparer()

Has Dagster changed its logging significantly in dagster==1.7.*? I'm noticing a lot more (not desirable) logging across the board.

gibsondan commented 3 weeks ago

Could https://github.com/dagster-io/dagster/pull/22267 (which went out in 1.7.10) have somehow caused this? The intention there was to change the formatting of warnings, not the frequency, but I’m wondering if it has unintended side effects.

I’m not sure how to square that with the problem occurring when you just upgrade dbt-core and don’t upgrade dagster though.

gibsondan commented 3 weeks ago

linking the similar report in https://github.com/dagster-io/dagster/discussions/22460 for context

NiallRees commented 3 weeks ago

Thanks for your quick response on this @gibsondan.

ssillaots-boku commented 3 weeks ago

Could https://github.com/dagster-io/dagster/pull/22267 (which went out in 1.7.10) have somehow caused this?

Don't know if this is question is targeted at me or Niall but from my point of view: doubt it since it started happening from Dagster 1.7.7