search

cimatosa / jobmanager

easy distributed computing based on the python class SyncManager for remote communication and python module multiprocessing for local parallelism
MIT License
6 stars 3 forks source link

Trying to write equivalent of mp.Pool().map, but failing with Python3 #22

Closed paulmueller closed 8 years ago

paulmueller commented 9 years ago

I am trying to create an equivalent of multiprocessing.Pool with the map functionality - only with a ProgressBar.

It seems to work with Python2, but I get an error with Python3:

_Pool_Server (24185): I'm the JobManager_Server main process
_Pool_Client (24186): FAILED to connect to (('localhost', 42524), 'map_0.601623858851451')
Process Process-6:
Traceback (most recent call last):
  File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
    self.run()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/paul/repos/jobmanager/jobmanager/decorators.py", line 113, in _run_jm_client
    client.start()
  File "/home/paul/repos/jobmanager/jobmanager/jobmanager.py", line 525, in start
    raise JMConnectionError("Can not start Client with no connection to server (shared objetcs are not available)")
ConnectionError: Can not start Client with no connection to server (shared objetcs are not available)
SyncManager (24188): started on X1Carbon:42524 with authkey 'bytearray(b'map_0.601623858851451')'
_Pool_Server (24185): started (host:X1Carbon authkey:map_0.601623858851451 port:42524 jobs:10)
_Pool_Server (24185): start processing incoming results
progress_bar (not started): terminal <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> was reserved for us <jobmanager.progress.ProgressBarFancy object at 0x7f70ef2dc780>
progress_bar (24190): started as new loop process
progress_bar (24190): setup signal handler for loop (SIGINT:ign, SIGTERM:ign)

The example script is there: https://github.com/cimatosa/jobmanager/blob/develop_pool/examples/pool_map.py

I implemented the Pool class in the decorators module: https://github.com/cimatosa/jobmanager/blob/develop_pool/jobmanager/decorators.py

paulmueller commented 8 years ago

Apparently there is a problem with the timing. I inserted a time.sleep(1) between start of server and creation of client. Then everything works.