nipype / pydra

Pydra Dataflow Engine
https://nipype.github.io/pydra/
Other
120 stars 59 forks source link

Provide wrapper for functions that modify inputs #625

Open effigies opened 1 year ago

effigies commented 1 year ago

In #623, @tclose added a function that modifies its input as a reliable reproducer of a task whose input hash changes after execution, resulting in a deadlocked workflow as the task is complete but its results are no longer findable in the cache directory.

In a comment I proposed an addition to the pydra.mark module:


If someone has an unsafe function, they're going to need to either rewrite or wrap it themselves, or we can provide a standard tool with straightforward semantics:

def wrap_unsafe(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        return func(*deepcopy(args), **deepcopy(kwargs))
    return wrapper

So we could write:

safe_task = task(wrap_unsafe(unsafe_func))

And if somebody has control over the function but is lazy or (more charitably) finds an explicit copy cluttering, they could write:

@task
@wrap_unsafe
def f(x):
    x.a = 2
    return x
tclose commented 1 year ago

In #623 you mentioned a check at the end of the task to see whether the task has altered its input hash. I think this is a good idea. Should that be included in this issue or in a separate one?

effigies commented 1 year ago

I think it makes sense as part of this issue. Good catch.