Multiprocessing support

[brunolnetto]

Hi,

Does it support multiprocessing? If not, this article may a good direction to follow: https://leimao.github.io/blog/Python-tqdm-Multiprocessing/

best regards

[TheTechRobo]

20

[brunolnetto]

Would you mind to design an example with the library ThreadingPoolExecutor available here?

[brunolnetto]

We may learn one or two tips with this library: https://github.com/alphatwirl/atpbar

[rsalmei]

Hey @brunolnetto! How are you?

Yes, it surely does support!

@TheTechRobo Actually, multiprocessing is very different than multiple bars... One can easily use multiprocessing or multithreading, maintaining one alive-progress bar on the screen, to monitor the completion of all those processes or threads.

I've made several complete examples of this kind of use already, please read #42, #75, and #100. The key is to use as_completed...

I'm going to close this, but feel free to reply with any other doubts. Regards! 👍

[brunolnetto]

My desire, mister feature waiter, is described on the issue from library below. I provide a CPU-inspired logic to implement it.

https://github.com/WoLpH/python-progressbar/issues/264#issuecomment-1025672569

The specific case for me is a multi-bar nested use-case. It is rather more complicated than nested fors since the parallel tasks may finish in different and unexpected orders from assignment.

Maybe you can provide this feature for users.

[brunolnetto]

I even answer your concern regarding multiple rate refresh with a reasonable approach.

[rsalmei]

So @TheTechRobo guessed right, you were indeed interested in #20, i.e. multiple bars instead of multiple processing. Well, unfortunately, it is very very hard to implement, and I do not have the time to tackle such a big endeavor in the foreseeable future... Perhaps someday...

[rsalmei]

These are the concerns: https://github.com/rsalmei/alive-progress/issues/20#issuecomment-605832026

[brunolnetto]

It seems line starve in combination to delete character or ANSI cursor move character may be useful here.

[TheTechRobo]

again, see the concerns

[brunolnetto]

I read and, again, it seems line starve in combination to delete character or ANSI cursor move character will be useful here. It means I already draw my strategy winner.

And also said again, my main strategy guidelines are below, rewritten from here.

The solution may lay on re-render the multiple bars. My best shot is:

• add a class property counter for the progress-bar instance
• listen to stout and look for \n character;
• update the current counter (for strongly typed languages, I recommend some either unsigned long int or scientific mantissa-like notation) for every \n on stout output;
• Bars with very different rates (100 or more) require special treatment. A good approach is to take, for example, the max of some default minimum refresh rate and either average or median of bars current rates.

Bars may always keep on the top of the terminal stack from the command run.

[rsalmei]

I know how to do it, it's how to implement it that is the problem.

Especially in alive-progress which, unlike other progress bars, has an active component, which renders the bar independently of the matter being monitored... My alive bar is multi-threaded, and achieves unmatched visual feedback of the current processing, making the live spinner run faster or slower with it. So it is not just "a cool spinner", it reacts to what you are doing in real-time.

Finally, making this engine work across multiple different processes, with possibly rather different velocities and timings, would be very challenging. I couldn't yet invest much time in it, especially without any kind of sponsorship or donations. I do this in my spare time, but perhaps someday, who knows 🙂