Progress Bar With ProcessExecutorPool

[cielowu]

Hi,

I'm writing an IO-consuming script with a progress bar and I'd like to improve its speed using a ProcessPoolExecutor.

I'm following your example on https://github.com/rsalmei/alive-progress/issues/75. However, since I'm new to multi-processing in Python and I'm facing some issues on making it work.

Here's my code sample:

def perform_query(query_end_time, resume_query_from=None):
    with open("processed_tasks.json", "r") as f:
        tasks = json.load(f)

    batch = 0
    if resume_query_from is not None:
        tasks = tasks[resume_query_from:]
        batch = resume_query_from / 1000

    total = len(tasks)
    with alive_bar(total) as bar, ProcessPoolExecutor(max_workers=2) as executor:
        try:
            start = 0
            while start < len(tasks):
                end = min(start + 1000, total)
                t = [(a, query_end_time) for a in tasks[start:end]]
                # perform_request is a simple requests call to an API and process the response
                futures = [executor.submit(perform_request, task) for task in t]
                query_responses = []
                for future in as_completed(futures):
                    bar()
                    query_responses.append(future.result())
                with open(f"processed_results_{batch}.json", "w") as f:
                    json.dump(query_responses, f)
                start += 1000
                batch += 1
        finally:
            executor.shutdown()

However, the above does not work, the progress bar is not updated.

[TheTechRobo]

potentially related: #42

[rsalmei]

Hello @cielowu,

It seems another process is printing to stdout, you have to avoid that. All the printing must be done through the main thread, the one where alive-progress is running.

Regarding the lack of progress, I think there wasn't any! We can see several stack traces popping up on the screen. To make the base infrastructure work, you should put together the bare minimum, make It work, then continue. Remove both the perform_request and the batch control, and you should see the problem clearly.

[rsalmei]

OK, going to close this one, as there isn't any problem here. Please continue if you do need any more help 👍

[cielowu]

Hi @rsalmei,

Thanks so much for the quick response. Thanks for your suggestion and I found several bugs I have in my code and it is working now.

[rsalmei]

Great @cielowu, you're welcome!