Parsl / parsl

Parsl - a Python parallel scripting library
http://parsl-project.org
Apache License 2.0
508 stars 196 forks source link

Dictionaries and tuples with `AppFuture`s do not get implicitly resolved #3108

Closed Andrew-S-Rosen closed 2 months ago

Andrew-S-Rosen commented 9 months ago

Describe the bug

When passing a dict or tuple (or other iterable) with an AppFuture in it as the input to a PythonApp, Parsl does not implicitly know to block and resolve it before proceeding with the next task.

import parsl
from parsl import python_app
from pathlib import Path

parsl.load()

@python_app
def job1():
    return Path.cwd()

@python_app
def job2(t: tuple[Path, str]):
    Path(t[0], t[1]).touch()

job2((job1(), "hello")).result()
import parsl
from parsl import python_app
from pathlib import Path

parsl.load()

@python_app
def job1():
    return Path.cwd()

@python_app
def job2(d: dict[Path, str]):
    for k,v in d.items():
        Path(k,v).touch()

job2({job1(): "hello"}).result()

In both scenarios, you will get the same TypeError: expected str, bytes or os.PathLike object, not AppFuture because the AppFuture is not automatically resolved by the time it gets to the Path() call.

Note to self that we should also confirm the following works once the simpler case is resolved: ```python import parsl from parsl import python_app from pathlib import Path parsl.load() @python_app def job1(): return {"directory": Path.cwd()} @python_app def job2(t: tuple[Path, str]): Path(t[0], t[1]).touch() job2((job1()["directory"], "hello")).result() ```
Andrew-S-Rosen commented 9 months ago

@yadudoc --- I'd be interested to get your thoughts on this one. It seems you have had similar ideas in prior issues, such as in https://github.com/Parsl/parsl/issues/1790#issuecomment-657710015. I would love to see at least some degree of Parsl-based introspection, and I'm curious what your perspective is.

Andrew-S-Rosen commented 9 months ago

My proposal would be the following: if an object __contains__ an AppFuture, then Parsl should know to implicitly block and resolve. This would cover the keys of dictionaries, and first level of lists and tuples, among other iterables. It would not be recursive, but I'd argue that this is better than nothing at all.

Andrew-S-Rosen commented 2 months ago

Closed in #3111.