dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.52k stars 1.45k forks source link

Display metadata from a Failure event on the asset info page #17752

Open NicolaiLolansen opened 11 months ago

NicolaiLolansen commented 11 months ago

What's the use case?

When an asset fails to materialize, there is currently no way to see that metadata on the asset info page, the information resides only in the run log. There is value in being able to see, for each partition, metadata surrounding the failed materialization. In my specific use case, I am copying files between S3 buckets and then I check metadata about the files and their contents. Ideally this could be done in a partition asset check, which is to my understanding on the roadmap, but not implemented yet: https://github.com/dagster-io/dagster/issues/17005

@jamiedemaria helped elucidate this for me here, in this discussion: https://github.com/dagster-io/dagster/discussions/17715

One issue, or bug maybe, I found is that if an asset has previously been successful in its materialization, the metadata from the previous materialization will stay if the asset then fails in a subsequent run, which can be really confusing if you're not aware of this.

Ideas of implementation

The Failure event already has metadata, and the workflow surrounding raising it from within an op/asset works as expected. I am not experienced in the Dagster stack, but I would imagine it is just this information, like the metadata field from the Output or context.add_output_metadata, that has to be added to the Asset Info Page.

Additional information

I would be willing to help implement this if guided in the right direction.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

jamiedemaria commented 11 months ago

thanks for raising this as an issue! I've categorized it so that it will get into the correct backlogs

jeanineharb commented 10 months ago

Interested in this feature as well!

AdrianTheopold commented 4 weeks ago

Interested as well!