search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.51k stars 1.45k forks source link

Show time elapsed by IOManager reading inputs / materialising outputs in run timeline #17891

Open mkleinbort opened 11 months ago

mkleinbort commented 11 months ago

What's the use case?

When looking at the timeline of an asset run, it's not readily apparent what time was spent computing a software defined asset vs. loading inputs / materialising outputs via the IOManager.

image

I propose a slight colour-coding of the timeline to indicate time spent in the IOManager

image

This can be helpful to debug at a glance when the asset code is slow vs, when the IOManager code is slow (eg. slow saving to remote location due to network issues, or database write limits, etc...)

Ideas of implementation

Just a slight colour change when the execution hands over from the IOManager to the asset code to the IOManager.

Additional information

I think I saw this issue somewhere, but can't find it - so adding it again.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

mkleinbort-ic commented 7 months ago

Any updates on this? It was also requested here: https://github.com/dagster-io/dagster/issues/15075

garethbrickman commented 7 months ago

@mkleinbort-ic No update as of yet, but the UI team have this in their review queue to assess for roadmapping!