JuliaFolds / Transducers.jl

Efficient transducers for Julia
https://juliafolds.github.io/Transducers.jl/dev/
MIT License
433 stars 24 forks source link

`withprogress` is barely helpful with threaded/distributed collectors #527

Open tecosaur opened 2 years ago

tecosaur commented 2 years ago

I'm a big fan of withprogress, when it works. It's nice and easy to slot it into a pipeline like so:

1:10000 |> withprogress |> Map(_ -> sleep(0.01)) |> collect

and I see a nice live-updating progress bar.

Unfortunately, the moment I start doing threaded or distributed computations it becomes markedly less helpful. With the example above, instead of smoothly updating in-step with the computation it has a tenancy to sit at 0% progress, and then jump to completion. This is particularly unhelpful when I have a multi-hour computation which I'd rather like to monitor.