search

swansonk14 / p_tqdm

Parallel processing with progress bars
MIT License
457 stars 44 forks source link

What could cause p_imap to hang on invocation? #18

Open zamirkhan opened 4 years ago

zamirkhan commented 4 years ago

Apologies for not having a small, reproducible snippet for this, but I'm using p_imap in my codebase which has been working fine for a while. However, recently (perhaps since me upgrading from 1.2?) I've had a couple of reproducible instances where my call to p_imap hangs. I print right before the call, and the function(s) that are invoked by p_imap also print (with flush=True) on the first line of those functions and it appears the functions never get invoked.

I'm passing a list of dictionaries to p_imap and the thing these examples have in common is that the list is longer and the dictionaries are larger. Is there some sort of size limitation on the parameters or is something else going on?

zamirkhan commented 4 years ago

As a follow-up, when I revert back to 1.2, this does not happen.

swansonk14 commented 4 years ago

Hi @zamirkhan, thanks for the question! I remember I ran into an issue like this with p_tqdm a long time ago and actually stopped using/developing p_tqdm for a little while because of it, but ever since coming back to p_tqdm I haven't run into the problem again and I never figured out what it was.

It's a little hard to debug without a reproducible snippet, but I'm curious, does the same problem happen if you swap out p_imap for p_map? I'm curious whether it's something specific to producing an iterator or if it's a general p_tqdm problem.

p_tqdm itself doesn't impose any sort of limitations of the sizes of the parameters. Under the hood I'm using the multiprocessing library pathos. As far as I know, pathos doesn't impose any limitations, but it is still possible that it's some other problem with the pathos library.

Please let me know what you find out when trying p_map instead of p_imap and let me know if you have any other insights that might help create a reproducible snippet so I can try to debug. I would love to get this issue resolved!