search

FAST-HEP / fasthep-flow

Convert YAML into a workflow DAG
Apache License 2.0
1 stars 1 forks source link

Add PythonOperator #35

Closed kreczko closed 1 month ago

kreczko commented 1 month ago

Fixes #12

ResultType currently does nothing, need to see how chaining functions will interact here before further implementation

YeungOnion commented 1 month ago

Rust has a pattern for handling results that are mutually exclusive success or failure, stderr poses some distinction here.

I got stuck in the, "how do I do this in a way that handles module aliases?" and then never got back to it. Can I ask how you test the example? I'm expecting to test with hatch

kreczko commented 1 month ago

how do I do this in a way that handles module aliases? Also got stuck here, not quite sure atm. Currently trying some things to get numpy work as np.

Can I ask how you test the example? Currently running

fasthep-flow execute docs/examples/hello_world.yaml

in a conda env (python 3.12). As this projects matures I will add tests to run with

pipx run nox -s tests

including the examples (hello world + CMS open data)

The result dict will likely change as I am looking at https://hamilton.dagworks.io/en/latest/reference/result-builders/Generic/ and the other variants. I would like something that is not dependent on hamilton, but can pass on enough metadata to convert correctly if hamilton is picked. Since awkward array has integrations with Dask, a custom ResultBuilder would be needed at some point.

Long story short, this project is still in alpha - trying to find its sweet-spot for implementation. Expect occasionally some incomplete functionality. That said, I finally have some time to push towards a beta that matches the current fast-flow release in terms of features.