Handling Execution Context and Parallelization

[ebolyen]

Discussion on how to handle execution context and parallelization in PyQi

Currently we are looking at IPython as a dependancy.

[josenavas]

@wasade

What do you think about including here the ClusterUtils outlined here? This way we can support Torque natively as well as IPython.

[wasade]

Would be much better to rely on IPython.parallel. If we go the route of supporting Torque natively and optionally IPython, then we effectively need an abstracted execution framework but I don't know what it buys us in this case as IPython can leverage Torque directly.

To be honest, those cluster_utils should actually use IPython.parallel instead of their current abuse but it was quicker to put those utilities in place at the time rather than the better solution.

[ebolyen]

It occurs to me that there are potentially two uses for parallelization withing PyQi:

1. Allowing multiple commands to be executed at the same time without blocking an interface.
2. Providing parallelization logic directly to commands: Imagine something like kwargs['pyqi']['parallel'].isMaster() and kwargs['pyqi']['parallel'].getEventBus()

However I don't know if 2 is worthwhile. Whatever support we would provide would still limit the methods available, so I am sure some commands would parallelize themselves anyways.

Thoughts?

EDIT: Also, the above reference is totally irrelevant to this conversation, but github insisted on linking.

[ebolyen]

Also to consider: How will we store execution ID's and state?

• Memory?
• Temp file?
• Database?
• A light key:value table?

I worry that there may be situations in which a memory based store would be GC'ed or at least no longer in scope unless we were to build in some sort of static singleton object that lived forever. (Which sounds like an anti-pattern if anything ever was.)