Might be worth highlighting that the default (Dask) executor parallelizes execution of electrons

[Andrew-S-Rosen]

What should we add?

Unless I've missed it somewhere in the docs, I'm not sure it's mentioned that the default (Dask but also Local) executor runs electrons in parallel when resources allow. This might be nice to highlight, as it demonstrates a clear value of dispatching workflows with Covalent even if you are just running them locally. This should be fairly obvious to anyone who has used Dask, but I imagine most users will be unaware of how Dask works and perhaps won't even realize that this is a feature they are getting out-of-the-box.

For instance, the following example is quite illustrative:

import covalent as ct
import time

@ct.electron
def add(a, b):
    time.sleep(5)
    return a + b

@ct.electron
def mult(a, b):
    time.sleep(5)
    return a * b

@ct.lattice
def workflow(a, b):
    out1 = add(a,b)
    out2 = mult(a,b)
    return out1 + out2

t_start = time.time()
dispatch_id = ct.dispatch(workflow)(1, 2)
result = ct.get_result(dispatch_id, wait=True)
print(time.time()-t_start)

The output time is < 10 seconds (I get 6.4 seconds). Naturally, the following returns in slightly over 10 s (with or without the decorators, of course).

import covalent as ct
import time

@ct.electron
def add(a, b):
    time.sleep(5)
    return a + b

@ct.electron
def mult(a, b):
    time.sleep(5)
    return a * b

@ct.lattice
def workflow(a, b):
    out1 = add(a,b)
    out2 = mult(a,b)
    return out1 + out2

t_start = time.time()
result = workflow(1, 2)
print(time.time()-t_start)

[Andrew-S-Rosen]

As a side-note, should executor="local" be parallelizing things?

[cjao]

Hi @Andrew-S-Rosen , executor="local" does indeed parallelize tasks using a simple ProcessPoolExecutor -- but you'll likely find Dask more reliable and capable. For instance, Dask allows one to cancel running tasks whereas with the "local" executor you can at most cancel tasks that haven't been picked up yet by the executor.

I would reserve the local executor for basic testing or debugging and use dask for anything more serious.

[santoshkumarradha]

Hi @Andrew-S-Rosen , executor="local" does indeed parallelize tasks using a simple ProcessPoolExecutor -- but you'll likely find Dask more reliable and capable. For instance, Dask allows one to cancel running tasks whereas with the "local" executor you can at most cancel tasks that haven't been picked up yet by the executor.

I would reserve the local executor for basic testing or debugging and use dask for anything more serious.

Also to add on to this, @Andrew-S-Rosen note that local does not schedule or handle memory requirements at all while dask will make sure to gracefully fail and allocate memory for each workers as well.