CMA-ES / pycma

Python implementation of CMA-ES
Other
1.12k stars 179 forks source link

Full cpu run in parallel raises exception #278

Open Landau1908 opened 2 weeks ago

Landau1908 commented 2 weeks ago

Hi, To get the maximum speedup, I used full cpu (multiprocessing.cpu_count()=64) to performing increasing popsize (IPOP). During which an exception is raised(see below codes). Does the number_of_processeshave to be set to es.popsize + 1 like the official example provided with EvalParallel2(elli, es.popsize + 1) as eval_all ?

Exception in thread Thread-7 (_handle_workers):
Traceback (most recent call last):
  File "d:\Users\fengyc\anaconda3\Lib\threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "d:\Users\fengyc\anaconda3\Lib\threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "d:\Users\fengyc\anaconda3\Lib\multiprocessing\pool.py", line 522, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "d:\Users\fengyc\anaconda3\Lib\multiprocessing\pool.py", line 502, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "d:\Users\fengyc\anaconda3\Lib\multiprocessing\connection.py", line 1084, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\Users\fengyc\anaconda3\Lib\multiprocessing\connection.py", line 1016, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: need at most 63 handles, got a sequence of length 66

Regards.

nikohansen commented 1 week ago

Does the number_of_processeshave to be set to es.popsize + 1 like the official example provided with EvalParallel2(elli, es.popsize + 1) as eval_all ?

No. It seems that the error suggests that the multiprocessing.cpu_count was not set to 64.