Open lucazav opened 3 years ago
@lucazav: I tested this on a mac, and it works for me. So, I'm going to assume that it's a windows issue. Python spawns processes on Windows differently then on other systems, and there are a few workarounds when things get stuck on Windows.
On Windows, you should generally use pathos.helpers.freeze_support()
, which requires a if __name__ == '__main__':
block of code. There's also multiprocess.set_start_method
to change the character of the Pool
, but I don't have a lot of experience with that function on windows, so I'm not sure if it's as functional as it is on a mac. I'd sit that aside for now. Going back to freeze_support
, if you find it throws an error once freeze_support
is added, then the next natural step would either be to use set_start_method
(to change how pools are created), or to use dill.settings['recurse'] = True
(to change how objects are serialized).
Give freeze_support
a try.
I'm trying to parallelize the row wise Pandas dataframe's
apply()
function, as I reported in this Stackoverflow question. Following the hint of albert, I run the following code using a conda environment with Python 3.9.1 64-bit on a Windows machine:Unfortunately it runs indefinitely on my machine using all the cores at 100%. Any hint?