Calling execute from code

[uatach]

Hello, is it possible to call an equivalent of jug execute from code? The context is that I want to configure tasks with values from other libs (e.g. one parameter would be the number of workers) and possibly use something like joblib to start jug workers. Thanks.

[luispedro]

You can naturally use functions like jug.jug.execution_loop from code, which is what jug execute does (or, for example, how it is used in tests). It probably requires that you know a bit about the internals of jug, though as it is not designed to be used this way and there is no well defined API for it.

I'm closing here as I feel this is not a bug or a feature request per se, but feel free to move the discussion to the mailing list: https://groups.google.com/d/forum/jug-users

[uatach]

Actually, after thinking a bit about it, I think this would be a nice feature... not sure it can be done already but we could replace this:

from math import sqrt
from joblib import Parallel, delayed
Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))

with this:

from jug import dispatch
from jugfile import is_odd, count, write

@dispatch(10, 'jugfile.py', '~/jugdir')
def tasks():
    # call tasks

if __name__ == '__main__':
    tasks()

then we could do this:

from jug import dispatch

@dispatch(10, 'jugfile.py', '~/jugdir')
def pipeline1():
    from jugfile import is_odd, count, write
    # call tasks

@dispatch(10, 'jugfile.py', '~/jugdir2')
def pipeline2():
    from jugfile import is_prime, count, write
    # call tasks

if __name__ == '__main__':
    pipeline1()
    pipeline2()