Issues with @jit and archipielago

[Fernando3161]

Hi! I am trying to use the jit decorator to increase speed in a large problem. I am testing with the following function:

`class toy_problem_o2:

def __init__(self, dim):
    self.dim = dim

def fitness(self, x):
    return [toy_problem_o2._a(x)[0], toy_problem_o2._b(x)[0], 
            -toy_problem_o2._c(x)[0]]

@jit(float64[:](float64[:]), nopython=True)
def _a(x):
    retval = np.zeros((1,))
    for x_i in x:
        retval[0]+=x_i
    return retval

@jit(float64[:](float64[:]), nopython=True)
def _b(x):
    retval = np.zeros((1,))
    sqr = np.zeros((1,))
    for x_i in x:
        sqr[0] += x_i*x_i
    retval[0]=1.-sqr[0]
    return retval

@jit(float64[:](float64[:]), nopython=True)
def _c(x):
    retval = np.zeros((1,))
    for x_i in x:
        retval[0]+=x_i
    return retval

def gradient(self, x):
    return pg.estimate_gradient(lambda x: self.fitness(x), x)  # numerical gradient

def get_nec(self):
    return 1

def get_nic(self):
    return 1

def get_bounds(self):
    return ([-1] * self.dim, [1] * self.dim)

def get_name(self):
    return "A toy problem, 2nd optimization"

def get_extra_info(self):
    return "\tDimensions: " + str(self.dim)`

`def archipielago_opt(f,n): start = time.time() funct=f(n) name=funct.get_name()

a_cstrs_sa = pg.algorithm(pg.cstrs_self_adaptive(iters=1000))
t1=time.time()
p_toy = pg.problem(funct)
p_toy.c_tol = [1e-4, 1e-4]
archi = pg.archipelago(n=16, algo=a_cstrs_sa, prob=p_toy, pop_size=10)
archi.evolve(2)
archi.wait_check()`

if __name__ == '__main__': archipielago_opt(toy_problem_o2,2) I am having an AssertionError,

RuntimeError: The asynchronous evolution of a pythonic island of type 'Multiprocessing island' raised an error: Traceback (most recent call last): File "D:\Programming\Anaconda3\lib\site-packages\pygmo_py_islands.py", line 225, in run_evolve ser_algo_pop = pickle.dumps((algo, pop)) File "D:\Programming\Anaconda3\lib\site-packages\cloudpickle\cloudpickle.py", line 1125, in dumps cp.dump(obj) File "D:\Programming\Anaconda3\lib\site-packages\cloudpickle\cloudpickle.py", line 482, in dump return Pickler.dump(self, obj) File "D:\Programming\Anaconda3\lib\pickle.py", line 437, in dump self.save(obj) File "D:\Programming\Anaconda3\lib\pickle.py", line 549, in save self.save_reduce(obj=obj, *rv) File "D:\Programming\Anaconda3\lib\pickle.py", line 633, in save_reduce save(cls) File "D:\Programming\Anaconda3\lib\pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "D:\Programming\Anaconda3\lib\site-packages\cloudpickle\cloudpickle.py", line 877, in save_global self.save_dynamic_class(obj) File "D:\Programming\Anaconda3\lib\site-packages\cloudpickle\cloudpickle.py", line 686, in save_dynamic_class save(clsdict) File "D:\Programming\Anaconda3\lib\pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "D:\Programming\Anaconda3\lib\pickle.py", line 859, in save_dict self._batch_setitems(obj.items()) File "D:\Programming\Anaconda3\lib\pickle.py", line 885, in _batch_setitems save(v) File "D:\Programming\Anaconda3\lib\pickle.py", line 524, in save rv = reduce(self.proto) File "D:\Programming\Anaconda3\lib\site-packages\numba\dispatcher.py", line 626, in reduce (self.class, str(self._uuid), File "D:\Programming\Anaconda3\lib\site-packages\numba\dispatcher.py", line 661, in _uuid self._set_uuid(u) File "D:\Programming\Anaconda3\lib\site-packages\numba\dispatcher.py", line 665, in _set_uuid assert self.__uuid is None AssertionError

When I do it without the decorator, the problem works fine and I get the theoretical optimal solutions for 2 variables.

Can it be solved with the jit decorator?

[Fernando3161]

https://github.com/Fernando3161/git_archi/blob/master/arch_example2.py

[bluescarni]

I think this is a problem related to the interaction between the jit decorator and pickling.

I'll try to test out the code tonight, but in the meantime a couple of things you could try:

• change the serialization backend in use by pygmo, see:

https://esa.github.io/pygmo2/misc.html?highlight=set_serialization#pygmo.set_serialization_backend

• try to put the jit decorator not on the methods directly, but on functions defined within the bodies of the methods. Something like:

def fitness(self, x):
    @jit(float64[:](float64[:]), nopython=True)
    def _a(x):
        ...

    @jit(float64[:](float64[:]), nopython=True)
    def _b(x):
        ...

    @jit(float64[:](float64[:]), nopython=True)
    def _c(x):
        ...

    return [_a(x)[0], _b(x)[0], _c(x)[0]]

[bluescarni]

@Fernando3161 I tried the arch_example2.py script on my linux machine and it does not produce any error. So perhaps this is a Windows-specific problem.

Did you try the suggestions I gave in my previous comment?

[bluescarni]

@Fernando3161 we had a similar problem in #31, where the workarounds suggested here successfully fixed the issue.

[Fernando3161]

Hello! The problem was solved. Thanks for the input. I could implement the decorators properly. Works in win10 https://github.com/Fernando3161/git_archi/blob/master/multi_objective_optmization.py

[bluescarni]

@Fernando3161 glad that you solved it!

Keep in mind that we are still experiencing some issues when numba is used in conjunction with multiprocessing. We haven't gotten to the bottom of it yet, but the problems seem confined to a specific linux setup currently.

Will close this in the meantime.