nalepae / pandarallel

A simple and efficient tool to parallelize Pandas operations on all available CPUs
https://nalepae.github.io/pandarallel
BSD 3-Clause "New" or "Revised" License
3.65k stars 210 forks source link

Control number of progress bars #243

Open applio opened 1 year ago

applio commented 1 year ago

As described in issue #242, this PR proposes adding a way to control the maximum number of progress bars to display that maintains backwards-compatibility with existing code.

Control over the number of progress bars is made through the existing input parameter to pandarallel.initialize():

pandarallel.pandarallel.initialize(nb_workers=42, progress_bar=5)

Expanded testing is provided by both expanding the number of params to the pytest.fixture, progress_bar() as well as one new specific unit test.

codecov[bot] commented 1 year ago

Codecov Report

Merging #243 (664f3ec) into master (46fc0e5) will decrease coverage by 0.96%. The diff coverage is 64.00%.

@@            Coverage Diff             @@
##           master     #243      +/-   ##
==========================================
- Coverage   91.37%   90.42%   -0.96%     
==========================================
  Files          12       12              
  Lines         580      595      +15     
==========================================
+ Hits          530      538       +8     
- Misses         50       57       +7     
Impacted Files Coverage Δ
pandarallel/progress_bars.py 72.79% <55.00%> (-2.62%) :arrow_down:
pandarallel/core.py 91.05% <100.00%> (+0.04%) :arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

applio commented 1 year ago

If I am interpreting the output from the codecov bot correctly, all of the decrease in the measured percentage of code coverage can be attributed to the patch's changes to code that only executes in a Jupyter notebook setting, specifically changes to ProgressBarsNotebookLab.__init__().

If desired, I could introduce mocks in an additional unittest to permit this to kind-of run just to get the code coverage percentage higher though I did not see any existing unit tests to do anything similar? I'm not sure how much practical value it would have beyond bumping the percentages?

nalepae commented 1 year ago

Thank you for your well-written PR!

However, I do not get exactly the point of having a number of progress bars that is neither 1 nor the number of workers.

I do understand that having like 128 progress bars is overwhelming. I do understand, in such a case, we simply would like to have only 1, global, progress bar.

However, I don't really see the added value to have something in between.

Can you please explain the rationale a little bit more?