Currently, it is hard to understand which step takes how much time, and there are some that take a lot (multiple minutes). It is important to know this, as some tasks take very long, and we need to understand what takes how long.
Note: To be done after #34 is finished.
Suggested solution 1
Maybe we can include this in the "housekeeping" we do for job status. So when the status of a job changes from x to y, we could do two log statements
[DEBUG] [Job 123] Finished 'x' in 6313 ms
[DEBUG] [Job 123] Starting stage 'y'
The benefit of this would be that it can also easliy be applied to other levels of granularity, so e.g. give an output how long a complete pipeline ran. And it could be easier to implement 🤵
Description
Currently, it is hard to understand which step takes how much time, and there are some that take a lot (multiple minutes). It is important to know this, as some tasks take very long, and we need to understand what takes how long.
Note: To be done after #34 is finished.
Suggested solution 1
Maybe we can include this in the "housekeeping" we do for job status. So when the status of a job changes from
x
toy
, we could do two log statementsSuggested solution 2
Use some kind of decorator.
The benefit of this would be that it can also easliy be applied to other levels of granularity, so e.g. give an output how long a complete pipeline ran. And it could be easier to implement 🤵