cisaacstern
commented
1 week ago @walljcg here's a start on how we can leverage lithops map reduce (as demonstrated in https://github.com/wildlife-dynamics/ecoscope-workflows/issues/28#issuecomment-2183694749) in the context of ecoscope workflows/tasks. The core task is here:
And here is an example of what it might look like compiled into a runnable workflow:
💡 Note that this does not yet run, but is more-so one step above pseudocode. Among other things, of course we don't actually need to split-apply-combine for time density maps in this way, I'm just using that as a toy example here.
In terms of where a script like this could run, if it runs locally, it can parallelize across python processes. In the cloud, we could package this script itself as a "launcher" serverless function on GCP Cloud Run, which once it gets to the lithops map-reduce section, would spawn additional Cloud Run functions. The "launcher" function (running this script) would then wait for all of the parallelized tasks to complete, gather the result, and send it back to wherever we need it (database api call to the server maybe).
Note also that this pattern (launch lithops from another cloud function) is conceptually almost identical to the Lithops Airflow Operator. (Which we may also want to use in the future for more complex DAGs, but "deploy lithops from a cloud function" is definitely lower latency for "easy/small" DAGs, as we've discussed.)
I've also iterated a bit on the workflow YAML spec design, so we can represent the map reduce operation there something like this:
To make this more legible, I'm experimenting with borrowing the GitHub Actions data structure of:
- name: "a human readable name"
id: |
# a unique id. we were using the task names for this before,
# but to support reuse of the same task within a workflow, we'll need an `id`
uses: # the action name in github, or for us, the task importable reference
from: |
# i'm not actually using this here, and i don't think it's part of GitHub Actions,
# but i thought this might be a nice way to include the github or pypi path for
# extension task packages, which could be dynamically installed in a venv at compile
# time (like pre-commit). https://github.com/moradology/venvception is a nice little
# package that does this (ephemeral venvs), that a former collaborator wrote for our
# last project.
with:
# kwargs to pass to the task
atmorling
Closes #28