Using multiprocessing pathos?

[kroscek]

Hi. I'm quite new when using multiprocessing. I want to try using pathos for multiprocessing per block of numpy array but it seems that your code is not callable by Pathos.

kf = simdkalman.KalmanFilter(
    state_transition,
    process_noise_cov,
    observation_model,
    observation_noise_cov)
with tqdm(total=train.as_matrix().shape[0], desc='  Encoding', unit='cols') as pbar:
    for n_rows in range(train.as_matrix().shape[0]):
        block=train.as_matrix()[n_rows:n_rows+100, :]
        result = kf.compute(block, 60)
        results = ProcessingPool().map(result, block)
        X.append(results.smoothed.observations.mean)
        pred.append(results.predicted.observations.mean)

TypeError: 'Result' object is not callable

[oseiskar]

Hi,

simdkalman should work just fine with Pathos.

It may help if you first familiarize yourself with using Pathos with a simpler example. The way you are using ProcessingPool's map is not quite correct. Unfortunately Pathos lacks online documentation, but the parallel map function is supposed to be called like p.map(function, data), for example

def multiplyByTwo(x):
    return x*2

p = ProcessPool()
p.map(multiplyByTwo, [1,2,3,4,5])

(see example here).

Also double-check the way you are splitting the full data to blocks.

[kroscek]

ah ok thanks!