swansonk14 / p_tqdm

Parallel processing with progress bars
MIT License
457 stars 44 forks source link

ValueError thrown if no iterables are sized #27

Closed cthoyt closed 2 years ago

cthoyt commented 4 years ago

The following code throws a ValueError:

from p_tqdm import p_uimap
def increment(x):
    return x + 1
it = (i for i in range(5))  # don't use range directly, because it is Sized
for x in p_uimap(f, it):
    print(x)

the error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/cthoyt/.virtualenvs/integrator/lib/python3.8/site-packages/p_tqdm/p_tqdm.py", line 42, in _parallel
    length = min(len(iterable) for iterable in iterables if isinstance(iterable, Sized))
ValueError: min() arg is an empty sequence

This happens because there are no sized iterables, and it's trying to take the min() of an empty sequence. This could be solved a few ways:

  1. Surrounding this line with try/except, then setting length=None. This is an optional argument to tqdm() so this is okay, but will not longer be able to give an estimate
try:
    # Determine length of tqdm (equal to length of shortest iterable)
    length = min(len(iterable) for iterable in iterables if isinstance(iterable, Sized))
except ValueError:
    length = None
  1. Save the list of iterables (this won't be so long) that are sized and check it explicitly for not being empty. If it is, set length=None
# Determine length of tqdm (equal to length of shortest iterable), if possible
lengths = [len(iterable) for iterable in iterables if isinstance(iterable, Sized)]
length = min(lengths) if lengths else None

I'm not sure which you would prefer, but they effectively accomplish the same thing. I made a PR #28 that uses the second solution.