uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

Getting results from pool.map() #184

Closed adurukan closed 4 years ago

adurukan commented 4 years ago

Hi, I have tried to create an issue of mine in the code below:

import numpy as np
from pathos.multiprocessing import ProcessPool
import multiprocessing
shapes_to_display=[]
products =[ 0,1,2,3,4,5,6,7,8,9]

def hay(i):
    print('PROCESSOR ID: ', multiprocessing.current_process().pid, 'Iteration: ', i)
    product = products[i]
    shapes_to_display.append([i, product])
    return shapes_to_display

pool = ProcessPool(nodes=4)
res= pool.map(hay, range(len(products)))

pool.close()
pool.restart()

res = np.asarray(res)
print(res.shape)

for r in res:
    r = np.asarray(r)
    print(r)

What I want to get as result is: (10,) [[0 0], [1,1], .. .. .. [9,9]]

What I end up getting: (10,) [[0 0]] [[1 1]] [[2 2]] [[3 3]] [[0 0] [4 4]] [[1 1] [5 5]] [[2 2] [6 6]] [[3 3] [7 7]] [[0 0] [4 4] [8 8]] [[1 1] [5 5] [9 9]] Can any of you help me to figure this out ? What do I need to do to get the result the way I want to ?

adurukan commented 4 years ago

Apparently appending to a list messes up since the processors are not aware of each other. Thus, I returned [i, shape] instead and that worked.