ipyparallel and pymoo

[patdub]

I am facing maximum recursing error problem when I try to use ipyparallel and pymoo. The error arrives when I send the class name (MyProblem) to the cluster. Do you have suggestions to not raise this type of error. Regards. Patrick.

[blankjul]

I have never used ipyparallel myself. Where exactly does it fail? Can you provide to reproduce it locally?

[patdub]

I have simplified the case for your investigation. Code is below:

import ipyparallel as ipp from pymoo.algorithms.moo.nsga2 import NSGA2 from pymoo.core.problem import ElementwiseProblem from pymoo.optimize import minimize

Very simplified problem

class MyProblem(ElementwiseProblem):

def __init__(self):
    super().__init__(n_var = 1,
                     n_obj = 1,
                     n_constr = 0,
                     xl=np.array([0]),
                     xu=np.array([1]))

def _evaluate(self, x, out, *args, **kwargs):

    # Fitness functions (values to minimize)
    f1 = -x[0]
    out["F"] = [f1] 

Start the clusters

nb_core = 16 rc = ipp.Cluster(n=nb_core).start_and_connect_sync() rc.wait_for_engines(n=nb_core) dview = rc[:]

Send class def to all cluster for further use

dview["MyProblem"] = MyProblem

When you run the code (either in Spyder or into a jupyterlab notebook, you have: RecursionError: maximum recursion depth exceeded

Thanks for your help. Patrick.

[blankjul]

Again, I don't know much about ipyparallel. But should it not be simply

dview["MyProblem"] = MyProblem()

Or why are you trying to push the class definition only?

[patdub]

Hello blankjul. I want to thank you very much. I try to add the parenthesis after the class name as you sugest and the error has gone. I did try yet to run the full program but this is promising. Regarding your last comment, I push the class defintion because it was unkown from the Cluster. Don't know why but I am also new in using multicore solution. Thank you again, after spending days to find a solution, you suggest me the right thing to do in few minutes. Patrick.

[patdub]

After verifications, the problem is still there. I have to send the class to the cluster (and not the instance) so the correct syntax is without the parenthesis at the end:

dview["MyProblem"] = MyProblem

and this leads to the recursing error :-(

[blankjul]

I could reproduce the error but it is simply not enough to get a clue what is going wrong. Can you create an issue in the ipyparallel framework and see if they can help? Are other parallelization frameworks such as dask a solution too?

[patdub]

Thanks Blankjul, I have also create the same issue in ipyparallel yesterday and I am waiting for their feedback. And yes I will investigate other parallelization solution like dask.

Keep you informed. Thanks again. Patrick.

[blankjul]

Great! I am going to close the ticket for now. Feel free to post when they are any updates though.