Displaying multiple bars at once

[blueberryberet]

Thanks for all your hard work, I think this is the best implementation of progress bars in python that I've come across so far. I'm trying to display multiple bars at once to represent independent torrent downloads. I've been able to achieve this using atpbar but when using alive_progress, I get the following error.

if sys.stdout.isatty() or config.force_tty: AttributeError: 'function' object has no attribute 'isatty'

Is there a good way to go about doing this? I'm not a developer, just trying this out for fun, so apologies for any glaring misunderstanding on my part.

[rsalmei]

Hey @blueberryberet, thank you for your kind words, I'm glad you liked it!

Unfortunately, it's not simple to support multithreading. The error you reported could be easily fixed, but I think the results would be unpredictable. My alive_bar controls the position of the cursor only horizontally, inside one line. I don't exactly know how you're doing that, but if all progress bars are in the same terminal, I suspect it won't work.

Would you have some test code?

[rsalmei]

As I'm sure you've seen, it is fixed in 1.3.3, but we're in unpredictable territory now. Please let me know what happens now, and if possible some test code to simulate here.

[blueberryberet]

Hi @rsalmei, using 1.3.3 did indeed fix my error and many thanks for that.

I've simplified my code below. The output is of both bars being drawn at the same cursor position, alternating between the two. I've tried several methods to try to draw them in different positions. One method was to call sys.stdout.write('\n') and sys.stdout.write('\x1b[1A') in between when each thread calls bar(). I also tried setting a global variable that would alternate between 0 and 1 each time each thread calls bar() to avoid any timing delays between threads. I also tried multiprocessing and calling alive_bar as a subprocess.

I note that increasing time.sleep() does not reduce the rate at which the bars are alternating between each other so maybe they are not refreshed/redrawn each time bar() is called?

Maybe using curses library might give an answer.

Despite your well documented code, I'm struggling to understand how the bars are `drawn.

import time
import sys
import threading
from alive_progress import alive_bar

def up():
    sys.stdout.write('\x1b[1A')
    sys.stdout.flush()

def down():
    sys.stdout.write('\n')
    sys.stdout.flush()

def task_1():
    global value
    items = range(1000)
    with alive_bar(len(items)) as bar:
        for item in items:
            while True:
                if value == 0:
                    down()
                    bar()
                    time.sleep(0.5)
                    value = 1

def task_2():
    global value
    items = range(2000)
    with alive_bar(len(items)) as bar:
        for item in items:
            while True:
                if value == 1:
                    up()
                    bar()
                    time.sleep(0.5)
                    value = 0

t1 = threading.Thread(target=task1)
t2 = threading.Thread(target=task2)

t1.start()
t2.start()

[blueberryberet]

[blueberryberet]

I was able to use escape codes to move the cursor as each bar was being drawn to solve the problem. I know this is a very hacky way of doing this but it works for my purposes! Thanks for your help and code!

def move (y, x): sys.__stdout__.write("\033[%d;%dH" % (y, x))

[luiztauffer]

@blueberryberet your solution in the GIF looks awesome! Exactly what I'm looking for, but I'm multiprocessing. Do you mind sharing the working code?

[blueberryberet]

Sure, no problem. It's been a while since I've looked at this and have never been good at commenting. I'll try to give a more detailed description when I have more time or let me know if you have a specific question, but here's the code.

When you run alive-progress in multiple threads it will try to render each bar on top of each other. I essentially added a move function to the progress.py module in rsalmei's code that redraws the cursor to a different row in the console each time alive_bar is called. I also passed two new arguments to progress.py, position and torrent_name. Above where update_data() is called in alive_repr function, I call the move function to draw the divider, torrent_name and bar at the various correct rows.

...
#FIRST CHANGE: the two added arguments 
def alive_bar(torrent_name, position, total=None, title=None, calibrate=None, **options):

    # SECOND CHANGE: the added function to move the cursor
    def move (y, x):
        sys.__stdout__.write("\033[%d;%dH" % (y, x))

...

    def alive_repr(spin=''):
        # THIRD CHNAGE: draw the added info and bar on the new rows
        move(position - 1, 1)
        sys.__stdout__.write('--------------------------------------------------------------')
        move(position, 1)
        sys.__stdout__.write(torrent_name)
        move(position + 1, 1)
        update_data()

This is a simplified version of my own python script that connects to qbittorrent and calls the alive-progress module using multithreading. Let me know if you want the full code. I'm not sure if your also using this to display torrent info or have a different idea. I'm also using the new windows terminal, you can get rid of the flashing cursor by editing the profiles.json file and setting the colour the same as your background

"cursorColor" : "#000000", ... ` import time import sys import threading from alive_progress import alive_bar

def task(torrent, x): global value items = range(1000) with alive_bar(torrent, (x+1)*4, len(items)) as bar: for item in items: bar() time.sleep(0.5)

torrent_list = [ 'First torrent', 'Second torrent', 'Third torrent' ] #etc, I used the qbittorrent api to get my torrent information x = 0 for torrent in torrent_list: thread = threading.Thread(target=task, args=(torrent, x)) time.sleep(0.1) thread.start() x = x + 1 `

[rsalmei]

Hello @blueberryberet, thanks for your ideias. I think I could implement something like that, I just don't like the position thing. I think the user should not have to say where the bar should appear, it should be automatic.

Also, if I were to implement that, I would have two more concerns:

1. each bar might be running at very different speeds, like one at 400k/s, and other at 20/s - the spinners would have to be updated at very different delays, so I'd need to choose one out of two strategies:
   1. update all bars when any of them needs a refresh (way simpler to control row position, but wastes processing to update slower bars without any need, and would have to think how to slow down spinners, like caching the last value, without asking for the next frame);
   2. or dynamically move the cursor row like you did, but without absolute positions, and with a proper Lock, to never "skip a beat", and suddenly updating in the wrong position - this seems to be better, but I'm afraid of the overhead introduced, with a new Lock and a few ifs to detect if the row should be moved, need to test it.
2. I think the various simultaneous bars would have to be in consecutive lines, one straight after the other, or the print hook would make no sense anymore; when the user code prints or logs anything, I need to adequately clean all bars on screen, print that content (how to enrich it with "the current position" if there were several?), and move the whole block of bars down... The system stdout is only one, I wouldn't be able to detect from where that print came.

What do you think? Any considerations??

[rsalmei]

Oh, forgot to say, in your GIF, it seems all the bars ended up leveling to the same speed, which would confirm my concern 1, that it is not easy to refresh at seemingly different speeds. Is it really the case?

[doug1as]

Hello @blueberryberet and @rsalmei ,

First, congratulations for provide to us this powerfull tool and tips.

My case is a bit different, your implementation "work" for me.

A doubt, is it possible to start the bars at the end/under of the last line printed/utilized of the terminal regardless of anything?

because I provide some information (print) before starting the bars and so they are overlapping my prints.

My scenario:

Terminal begin/first line - Call my script.py type my login for script <---------------------------------------Progress Bars are starting here for example and overlapping type my password for script <---------------------------------------And so on /n /n print my information thread 1--------- /n /n print my information thread 2--------- /n <---------------------------------------Progress Bars are starting here for example and overlapping (if i change start position variable from 0 to 'n', 11 for example) /n <---------------------------------------And so on print my information thread "n"---------

My need:

Terminal begin/first line - Call my script.py type my login for script type my password for script /n /n print my information thread 1--------- /n /n print my information thread 2--------- /n /n print my information thread "n"--------- /n /n new line/blank line<---------------------------------------First Progress needs to start here for example new line/blank line<---------------------------------------Second Progress needs to start here for example new line/blank line<---------------------------------------And so on

And if i need to expand out the last line, no scrooll appear, new bars its overlapping aggain (i need to show +- 500 simultaneosly progress bars bars)

Regards

[yurenchen000]

@doug1as hey, I found this progress seems works as you need.

https://rich.readthedocs.io/en/stable/progress.html#advanced-usage

https://user-images.githubusercontent.com/8458213/144310341-96d69049-5b57-432e-a4dc-c564edf7d3f7.mp4

test code:

import time

from rich.progress import Progress

with Progress() as progress:

    print('hello1')
    task1 = progress.add_task("[red]Downloading...", total=1000)
    task2 = progress.add_task("[green]Processing...", total=1000)
    task3 = progress.add_task("[cyan]Cooking...", total=1000)

    print('hello2')
    i = 0
    while not progress.finished:
        progress.update(task1, advance=0.5)
        progress.update(task2, advance=0.3)
        progress.update(task3, advance=0.9)
        i+=1
        print(f'hello {i}')
        time.sleep(0.02)

[doug1as]

hey, I found this progress seems works as you need.

I'll try to implement it here.

Thanks in advance!

[Wamy-Dev]

This is desperately needed, I am trying to run about 15 at once and it just prints them over each other.

[rsalmei]

Yeah, I can imagine... Please share your code. It's good to see how people imagine themselves using a feature like this, helps me figure out what kind of interface it would be required (one right below each other? y coordinates? both?) I have little free time nowadays, and tbh I'm more interested in Rust now, but who knows! I'm very proud of what I've already achieved here.

[245851712]

Hi @rsalmei, I want to run many processes by calling the same function with different param in parallel, and show how many processe have been done like this. I want to know how can i implement that,thank you!

from multiprocessing import Pool
import time
def test(p,):
       print(p)

       time.sleep(3)
if __name__=="__main__":
    pool =  #Pool(processes=9)
    for i  in range(7724):    
        pool.apply_async(test, args=(i,))   

    print('test')
    pool.close()

    pool.join()

[rsalmei]

Hi @245851712, you are on the wrong thread, you are not trying to display multiple bars at once.

Anyway, search the past tickets here for "pool", you'll find several examples of doing what you need. On a quick search I've found this and this, which might help you.

[discobot]

Hi! @rsalmei, I think I have a relevant example: Suppose that I have a function that does computation over a file.

def process(fname):
    with open(fname, 'r') as f:
       for line in  f:
           do_work(line)

and than I call this function through multiprocessing, like this:

 tasks = [...]
 with Pool(processes=8) as p:
          p.map(process, tasks)

Ideally I want a progressbars both for the inner loops (which are len(tasks) in total and 8 at any given time) and an outer one for the global progress.