search

dattalab / pyhsmm-library-models

library models built on top of pyhsmm
0 stars 1 forks source link

Flexible IPython parallel profile usage #49

Open alexbw opened 10 years ago

alexbw commented 10 years ago

Right now, we have to hand-code which IPython parallel profile we're working on. This becomes troublesome as we jump between machines, and it makes certain types of parallelization impossible on shared-filesystem compute clusters.

Here's the changes in pyhsmm-library-models

And here are the changes in pyhsmm

The only extra work on the user of the new parallel code is to specify which parallel profile should be used. I think it is good to specify the profile, especially on the architectures we're jumping between. Thoughts?

mattjj commented 10 years ago

What were your motivations for putting some things in a singleton class (ParallelState) instead of keeping everything in the module scope?

alexbw commented 10 years ago

pyhsmm.parallel can be imported multiple times in the code we use, and what I found is that state is not preserved for module-scope definitions. So, I used the environment variables of the system to store some state, and make it accessible from any part in a codebase running on a machine.

mattjj commented 10 years ago

Environment variables are a separate issue; I'm asking about the class.

The class organization won't behave any differently from the old code with respect to multiple imports: it's instantiated once at the module level (parallel_state = ParallelState()) just like the old code set up its global state at the module level. And importing multiple times doesn't actually re-load the module:

A module can contain executable statements as well as function definitions. These statements are intended to initialize the module. They are executed only the first time the module name is encountered in an import statement.

http://docs.python.org/2/tutorial/modules.html#more-on-modules

Import statements are executed in two steps: (1) find a module, and initialize it if necessary; (2) define a name or names in the local namespace (of the scope where the import statement occurs).

http://docs.python.org/2/reference/simple_stmts.html#the-import-statement

Modules on their own basically act like singleton classes, so I'm wondering if there is a motivation I'm not thinking of here.

mattjj commented 10 years ago

Did you notice incorrect behavior in some use case? (With the module-level state, not regarding environment variables.)