Bars not getting removed, with leave=False in threaded program

[bunjiboys]

I'm trying to figure out why my bars aren't getting removed after they finish, when the updates are processed using threads.

Scenario: I need to process a list of files and wanted to have it show first an overall progress bar showing the total progress, and then a bar for each file being processed with the processing of the files happening in a separate thread for parallelization. The bars get updated just fine, but when I call bar.close() the bars are not removed from the manager.

Example code:

import random
import string
import time
from concurrent.futures import TimeoutError as FutureTimeoutError
from concurrent.futures.thread import ThreadPoolExecutor

import enlighten

manager = enlighten.get_manager()

def worker(progress_bar):
    while progress_bar.count < progress_bar.total:
        time.sleep(random.randint(0, 3) / 10)
        progress_bar.update(incr=1, force=True)
    progress_bar.close()

bars = []
for idx in range(random.randint(3, 10)):
    bars.append({
        'desc': string.ascii_lowercase[idx],
        'total': random.randint(25, 150)
    })

overall = manager.counter(total=len(bars), desc='Overall')
futures = []
with ThreadPoolExecutor(max_workers=3) as executor:
    while bars:
        bar_info = bars.pop()
        bar = manager.counter(total=bar_info['total'], desc=bar_info['desc'], leave=False)
        futures.append(executor.submit(worker, bar))

    while futures:
        for future in futures:
            try:
                future.result(timeout=.1)
                overall.update(incr=1, force=True)
                futures.remove(future)
            except FutureTimeoutError:
                pass

I have another part of the program I'm working on, where I process these files serially and in that case the bars are removed just fine, so it seems to me this might be something specific to when you're handling it with threads.

[avylove]

Thanks for reporting your issue! Yes, I think it's the threading. I'm not sure why exactly though.

I do recommend, for threading and multiprocessing, to keep all the progress bars in one thread/process, preferably the main process. I have an similar multiprocessing example here. It uses queues and the parent process iterates through the queues periodically to check for updates. In my case, I'm not adding the progress bar until the subprocess starts, which keeps the number displayed down to only what is being processed.

[avylove]

Were you able to implement a workaround?

I was thinking about how to make Enlighten more thread safe. I'm still not sure if it's the right direction or not, but it could probably be accomplished by putting mutexes on most of the methods in Manager. Thoughts?

[bunjiboys]

Sorry for the delay, yeah I got it working using queues to push data from child threads to the parent, which fixed the issue