search

lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

Make progress bar separate from log level #1328

Closed rabernat closed 5 months ago

rabernat commented 5 months ago

I am running map jobs with Lithops with O(100000) tasks. I greatly appreciate the ability to have a progress bar to monitor completion.

However, currently, the progress bar only shows up if the log level is exactly equal to INFO (not less or not more):

https://github.com/lithops-cloud/lithops/blob/033e725b320998ae85e4e763a87169f9028b42be/lithops/wait.py#L116-L117

The problem here is that INFO also prints out a message for every single task I start:

2024-05-02 16:37:34,528 [INFO] invokers.py:174 -- ExecutorID 55eecc-0 | JobID A006 - Starting function invocation: process_chunk() - Total: 1 activations

That means if I want to have a progress bar for my 100000 tasks, I also need to accept 100000 lines being printed to my console. That is definitely NOT what I want.

I propose to simply remove the log-level check from the code cited above and make progress bar completely separate from log level.

gilv commented 5 months ago

@JosepSampe will you be able to look into this?

JosepSampe commented 5 months ago

The idea behind this was not to show the progress bar during DEBUG logs, as DEBUG logs show a lot of lines, and print the progress bar is a bit messy. I agree in your case is also a bit messy, so we can remove the logger.getEffectiveLevel() == logging.INFO check as you did in the PR. In addition, IMO, I would automatically disable the progress bar if logs are DEBUG to keep the same behavior we had up until now.