When an asset fails to materialize, there is currently no way to see that metadata on the asset info page, the information resides only in the run log. There is value in being able to see, for each partition, metadata surrounding the failed materialization. In my specific use case, I am copying files between S3 buckets and then I check metadata about the files and their contents. Ideally this could be done in a partition asset check, which is to my understanding on the roadmap, but not implemented yet: https://github.com/dagster-io/dagster/issues/17005
One issue, or bug maybe, I found is that if an asset has previously been successful in its materialization, the metadata from the previous materialization will stay if the asset then fails in a subsequent run, which can be really confusing if you're not aware of this.
Ideas of implementation
The Failure event already has metadata, and the workflow surrounding raising it from within an op/asset works as expected. I am not experienced in the Dagster stack, but I would imagine it is just this information, like the metadata field from the Output or context.add_output_metadata, that has to be added to the Asset Info Page.
Additional information
I would be willing to help implement this if guided in the right direction.
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
What's the use case?
When an asset fails to materialize, there is currently no way to see that metadata on the asset info page, the information resides only in the run log. There is value in being able to see, for each partition, metadata surrounding the failed materialization. In my specific use case, I am copying files between S3 buckets and then I check metadata about the files and their contents. Ideally this could be done in a partition asset check, which is to my understanding on the roadmap, but not implemented yet: https://github.com/dagster-io/dagster/issues/17005
@jamiedemaria helped elucidate this for me here, in this discussion: https://github.com/dagster-io/dagster/discussions/17715
One issue, or bug maybe, I found is that if an asset has previously been successful in its materialization, the metadata from the previous materialization will stay if the asset then fails in a subsequent run, which can be really confusing if you're not aware of this.
Ideas of implementation
The
Failure
event already has metadata, and the workflow surrounding raising it from within an op/asset works as expected. I am not experienced in the Dagster stack, but I would imagine it is just this information, like the metadata field from theOutput
orcontext.add_output_metadata
, that has to be added to the Asset Info Page.Additional information
I would be willing to help implement this if guided in the right direction.
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.