dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
12.01k stars 1.5k forks source link

Asset daemon crashes when auto-materializing TimeWindow asset with future start_date #17191

Open psarka opened 1 year ago

psarka commented 1 year ago

Dagster version

1.5.3

What's the issue?

Stack trace:

  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_core/definitions/asset_daemon_context.py", line 548, in evaluate
    evaluations, to_materialize, to_discard = self.get_auto_materialize_asset_evaluations()
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_core/definitions/asset_daemon_context.py", line 471, in get_auto_materialize_asset_evaluations
    (evaluation, to_materialize_for_asset, to_discard_for_asset) = self.evaluate_asset(
                                                                   ^^^^^^^^^^^^^^^^^^^^
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_core/definitions/asset_daemon_context.py", line 337, in evaluate_asset
    for evaluation_data, asset_partitions in materialize_rule.evaluate_for_asset(
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_core/definitions/auto_materialize_rule.py", line 416, in evaluate_for_asset
    context.daemon_context.get_never_handled_root_asset_partitions_for_key(
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_core/definitions/asset_daemon_context.py", line 224, in get_never_handled_root_asset_partitions_for_key
    never_handled, _, _ = self._get_never_handled_and_newly_handled_root_asset_partitions()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_utils/cached_method.py", line 66, in _cached_method_wrapper
    result = method(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_core/definitions/asset_daemon_context.py", line 187, in _get_never_handled_and_newly_handled_root_asset_partitions
    for partition_key in self.cursor.get_unhandled_partitions(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_core/definitions/asset_daemon_cursor.py", line 76, in get_unhandled_partitions
    return handled_subset.get_partition_keys_not_in_subset(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_core/definitions/time_window_partitions.py", line 1459, in get_partition_keys_not_in_subset
    for tw in self._get_partition_time_windows_not_in_subset(current_time):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_core/definitions/time_window_partitions.py", line 1420, in _get_partition_time_windows_not_in_subset
    check.failed("No partitions found")
  File "/home/psarka/mambaforge/envs/repro/lib/python3.11/site-packages/dagster/_check/__init__.py", line 1590, in failed
    raise CheckError(f"Failure condition: {desc}")

What did you expect to happen?

Not crash

How to reproduce?

Automaterialize this asset

from dagster import AutoMaterializePolicy, asset, DailyPartitionsDefinition

@asset(
    name='x',
    partitions_def=DailyPartitionsDefinition(start_date="2024-01-01"),
    auto_materialize_policy=AutoMaterializePolicy.eager(1000),
)
def x() -> str:
    return 'hi'

Deployment type

None

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

OwenKephart commented 1 year ago

Hi @psarka -- thanks for the report, I put up a PR to resolve this.