search

datajoely / kedro-rich

Make your Kedro experience snazzy
36 stars 4 forks source link

`ParallelRunner` doesn't work yet #4

Open datajoely opened 2 years ago

datajoely commented 2 years ago

Description

Due to the way the ParallelRunner works in Kedro via multi-processing the existing hook implementation breaks when interacting with whatever UI thread rich needs to work.

Context

Currently if the user specifies parallel runner the progress bar logic is disabled.

Potential solutions to evaluate:

  1. Work out a way of making the existing implementation using Locks (preferred)
  2. Subclass the ParallelRunner implementation and do it there (not ideal)
datajoely commented 2 years ago

I can't get the progress bars to work with ParallelRunner, but the nicer logging experience still works. Ultimately without developing my own runner implementation, I think the the interference of the threading rich is using behind the scenes and the multiprocessing Kedro is doing just won't work nicely together.