search

phanein / deepwalk

DeepWalk - Deep Learning for Graphs
http://www.perozzi.net/projects/deepwalk/
Other
2.68k stars 825 forks source link

concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending #95

Open fenglonz opened 5 years ago

fenglonz commented 5 years ago

Traceback (most recent call last): File "./deepwalk", line 11, in load_entry_point('deepwalk==1.0.3', 'console_scripts', 'deepwalk')() File "/usr/local/python36/lib/python3.6/site-packages/deepwalk-1.0.3-py3.6.egg/deepwalk/main.py", line 162, in main process(args) File "/usr/local/python36/lib/python3.6/site-packages/deepwalk-1.0.3-py3.6.egg/deepwalk/main.py", line 83, in process num_workers=args.workers) File "/usr/local/python36/lib/python3.6/site-packages/deepwalk-1.0.3-py3.6.egg/deepwalk/walks.py", line 85, in write_walks_todisk for file in executor.map(_write_walks_to_disk, args_list): File "/usr/local/python36/lib/python3.6/concurrent/futures/process.py", line 366, in _chain_from_iterable_of_lists for element in iterable: File "/usr/local/python36/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/local/python36/lib/python3.6/concurrent/futures/_base.py", line 432, in result return self.get_result() File "/usr/local/python36/lib/python3.6/concurrent/futures/_base.py", line 384, in get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

fenglonz commented 5 years ago

84 with ProcessPoolExecutor(max_workers=numworkers) as executor: 85 for file in executor.map(_write_walks_to_disk, argslist): 86 files.append(file)

It fails at walks.py:85. How to fix?

GTmac commented 5 years ago

What OS are you using? What is the exact command you ran? This question seems to be related: https://stackoverflow.com/questions/15900366/all-example-concurrent-futures-code-is-failing-with-brokenprocesspool

NZiozis commented 3 years ago

Yes, I too believe that it is related to the issue that you linked. The way that I run into this problem is when I try to run multiple workers, but can't keep all the walks in memory. So, when I try to run the following command on a large graph, it crashed with the original issue message above. I am using Debian Buster.

--number-walks 80 --representation-size 128 --walk-length 40 --window-size 10 --workers 20

Have you found a solution to this bug, or have any idea about how I can go about fixing it. I am not so familiar with the python ProcessPoolExecutor library, and as indicated by the article you linked the error messages aren't very helpful.