parallel electrode warping

[bendichter]

Electrode warping can be massively sped up by parallelization. We should probably use submit_job for warps on the server and multiprocessing.pool for warps on individual computers.

[libertyh]

Do you have a version that does this? I'm not too familiar with multiprocessing.pool so if you have an example that would be excellent. We should also have a way of limiting the number of simultaneous processes to limit crashes :)

[bendichter]

Yeah that's precisely what pool does

[bendichter]

import multiprocessing
import subprocess

def work(cmd):
    return subprocess.call(cmd, shell=False)

count = multiprocessing.cpu_count()
pool = multiprocessing.Pool(processes=count)
print pool.map(work, list_of_commands)

[bendichter]

though I was thinking it might be better to take everything inside of for c in np.arange(len(surface_indices)): and put it into the a function called warp_single_electrode and have that be the work function. Then use partial to fill in the arguments of warp_single_electrode except for c and then call pool.map(warp_single_electrode, np.arange(len(surface_indices)))

[libertyh]

That makes sense -- have every electrode warp as a single "job". What is partial here?

[bendichter]

partial allows you to fill in optional arguments of a function. In this case, you I think you can only pass one argument in pool.map, so you could use partial to pass all of the constant variables like hem, template, etc.