dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.69k stars 1.48k forks source link

Time partitioned self-dependent assets do not automaterialize when descending from unpartitioned asset #16890

Open psarka opened 1 year ago

psarka commented 1 year ago

Dagster version

1.4.16

What's the issue?

As stated in the title.

image

(pinging @OwenKephart, this is the issue as you suggested).

What did you expect to happen?

Self-dependent time partitioned assets automaterialize correctly (in my view) when they are root assets. They automaterialize sequentially (partition by partition) and the most recent materializations_per_minute partitions are materialized. That is:

I expect the same to hold when self-dependent time partitioned assets descend from unpartitioned asset.

How to reproduce?

repro here: https://github.com/psarka/repro/tree/auto-time-window-2

Deployment type

None

Deployment details

No response

Additional information

Supersedes https://github.com/dagster-io/dagster/pull/16646 Supersedes https://github.com/dagster-io/dagster/issues/16033

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

littleK0i commented 11 months ago

I was about to report the same bug. It still exists in the current version 1.5.11, and it took a while to figure out.

Graph:

Screenshot 2023-12-11 at 18 11 57

Auto-materialization log:

Screenshot 2023-12-11 at 18 12 16

Details:

Screenshot 2023-12-11 at 18 12 34

Asset seems to be waiting for itself indefinitely.