Introduce rich progress bars to the `SequentialRunner`

antonymilne commented 1 year ago

Add rich progress bars for the SequentialRunner only. Adding it for the ParallelRunner is a separate task.

Originally written by @datajoely in #1464.

Today the only way to track runner progress is via the log output
The rich.logging.RichHandler allows us to introduce a complimentary set of rich.progress.Progress bars which will render nicely.
I've managed to get this sort of working via hooks in my plug-in.
However, I've struggled to get these working with ParallelRunner. Multiprocessing is weird since it's hard to share memory.
From a plug-in point of view I think it's fine to say that this isn't supported, but if we were doing this natively we could do so directly within the runner or potentially introduce a set of runner hooks that would allow us to manage state better.
A really cool end state would be to have a mini-progress bar for each ParallelRunner sub-branch 🚀

merelcht commented 2 weeks ago

@astrojuanlu Shall we just close all the issues about adding more rich stuff?

astrojuanlu commented 2 weeks ago

As much as I'd like fancy progress bars in the Kedro output, I agree we shouldn't do this until we've untangled Kedro's dependency on rich. So maybe we can tackle this in 2 phases: de-rich (things like https://github.com/kedro-org/kedro/issues/3985, https://github.com/kedro-org/kedro/issues/3967#issuecomment-2209369563, essentially this milestone) and then re-introduce more interactivity on the CLI at a later stage (Kedro 0.20.x...?)

kedro-org / kedro