Performance research: run independent analyses in parallel

[rochoa]

With the current approach, we are limited to run one analysis at a time per user database, due to the way camshaft and batch queries collaborate.

Although in most of the cases that's more than enough as a user must be doing one analysis pipeline at a time. However, when a user has independent analyses the same limitation stands.

With a scenario as:

A --------> D
B ---> C ---^

With the current approach, we run the analyses like:

t0:A, t1:B, t2:C, t3:D

However, we should be able to run them like:

t0:A, t1:C, t2:D
t0:B

To accomplish this, we need to change the way camshaft and batch queries collaborate and enable them to be able to run more than one query at a time.

[jgoizueta]

Some general ideas for discussion (maybe flawed due to my lack of knowledge of the SQL API)

We need to add some kind of paralallelization interface to the SQL (batch) API.

I think the most simple and effective way would be for that interface to accept a DAG of tasks (SQL queries), rather than using any other kind of primitives to manage parallelism.

The client, camshaft, already has the analysis in DAG form, and that allows for efficient and simple parallelization; given a maximum paralellism level of N (execution slots), and considering a node as runnable if its dependency free:

• Take up to N runnable nodes and schedule them for parallel execution
• When a node task finishes, the corresponding node is removed from the DAG (so other nodes can become free of dependencies, i.e. runnable), then we can schedule runnable nodes to free execution slots until all nodes are executed