search

vmlaker / mpipe

Python API for writing multiprocessing pipelines
http://vmlaker.github.io/mpipe
MIT License
86 stars 25 forks source link

Basic Example broken. #12

Open poleguy opened 7 years ago

poleguy commented 7 years ago

I'm trying this for the first time, and the simple example on the main page does not work: http://vmlaker.github.io/mpipe/

Tried on python 3.4.5 and 3.6.1 Windows 7 Anaconda 4.4.0 64-bit

Has this ever been tested on windows?

It errors out with: ... File "C:\Anaconda3\lib\site-packages\mpipe\Pipeline.py", line 9, in init self._input_stage.build() File "C:\Anaconda3\lib\site-packages\mpipe\Stage.py", line 99, in build self._do_stop_task, File "C:\Anaconda3\lib\site-packages\mpipe\OrderedWorker.py", line 93, in assemble worker.start() File "C:\Anaconda3\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "C:\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child) File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'OrderedStage.init..wclass'

vmlaker commented 7 years ago

Sorry, hasn't been tested on Windows. Are you running this from the Python console? Try saving the code to a file and run with command:

python test.py

anviloro commented 6 years ago

same here, tried on windows didn't work, tried on ubuntu server, worked like a charm.

AndrewMartchenko commented 5 years ago

Hi, I think I have a solution to this problem. This problem is occurring because "wclass" class is dynamically defined inside the OrderedStage and UnorderedStage constructor. Python uses lambda magic to create this class and so it is not picklable in windows. To solve the problem you will need to pass the "target" function into the "wclass" constructor and possible define "wclass" globally. Hope this helps.

Edit: In windows functions can only be pickled if they are defined in the top level. So you will have to globally define 'wclass' for multiprocessing to pickle it.