Open emlys opened 11 months ago
Related to #83
We talked about this today and decided to wait until after the taskgraph 1.0 release. Taskgraph has been stable for a while now, and we want to do the release before mixing anything up.
There might be more minimal ways to add support for lambdas, like patching multiprocessing
without using multiprocess
, or passing in lambdas as strings.
Callables passed into
TaskGraph.add_task
currently have to be named functions defined in the global scope. There are many cases where it would be convenient to pass in a lambda function defined in-line. I'm specifically thinking aboutraster_map
, e.g.It would be convenient to write
op=lambda x: ...
, but that breaks whenn_workers > 0
because the args must be pickled. Andpickle
cannot pickle lambdas or local objects.Taskgraph could support lambdas and local callables by using
multiprocess
, a fork of python'smultiprocessing
that supports pickling more types. I briefly tried replacingmultiprocessing
withmultiprocess
in Task.py and it worked (test suite passed and was able to pass lambdas into taskgraph). But there could be other implications of usingmultiprocess
, like conflicts withmultiprocessing
if both were used.It's also worth noting that, while python supports pickling
functools.partial
s, taskgraph raises an error because it relies on the__name__
attribute. It would be nice to supportpartial
s too.