scidash / neuronunit

A package for data-driven validation of neuron and ion channel models using SciUnit
http://neuronunit.scidash.org
38 stars 24 forks source link

How to create unique integers label that label CPUs that can be used as global identifiers (gids). #135

Closed russelljjarvis closed 6 years ago

russelljjarvis commented 6 years ago

https://github.com/ipython/ipyparallel/blob/master/examples/itermapresult.py#L32-L36 I have tried this, and it does not quiet work as it should.

import ipyparallel as ipp
# create client & view
rc = ipp.Client(profile='default')

dv = rc[:]

# scatter 'id', so id=0,1,2 on engines 0,1,2
dv.scatter('id', rc.ids, flatten=True)
print("Engine IDs: ", dv['id'])
# create a Reference to `id`. This will be a different value on each engine
ref = ipp.Reference('id')

dview.apply(print,ref)

russelljjarvis commented 6 years ago

The way to use the reference is to apply it on the workers:

dview.apply(print,ref)

yields:


<AsyncResult: print>
(Pdb) engine.0.stdout: 0

engine.5.stdout: 5

engine.6.stdout: 6

engine.7.stdout: 7

engine.3.stdout: 3

engine.2.stdout: 2

engine.4.stdout: 4

engine.1.stdout: 1

I believe the way to do parallel safe file writing, is to make temp files whose names are dependent on unique CPU labels, since each CPU writes to a different temp file, none of the CPUs can corrupt the others files. The only problem is file reads would have to depend on the same operation.