wildlife-dynamics / ecoscope-workflows

An extensible task specification and compiler for local and distributed workflows.
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Rework compilation spec schema to look more like GitHub Workflows #50

Open cisaacstern opened 5 days ago

cisaacstern commented 5 days ago

The current compilation spec schema is represented by this example:

https://github.com/wildlife-dynamics/ecoscope-workflows/blob/08a8e05f25c2824a987effc0effbf47f59d3d0fe/examples/compilation-specs/time-density.yaml#L1-L12

This has worked so far but is not expressive enough to capture the full range of relationships we will want to encode in these specs. As first noted in https://github.com/wildlife-dynamics/ecoscope-workflows/pull/45#issuecomment-2185513124, I believe we will be better served by adapting a schema inspired by GitHub Workflows jobs.step, i.e.

- name: "a human readable name"
  id: |
    # a unique id. we were using the task names for this before,
    # but to support reuse of the same task within a workflow, we'll need an `id`
  uses:  # the action name in github, or for us, the task importable reference
  from: |
    # i'm not actually using this here, and i don't think it's part of GitHub Actions,
    # but i thought this might be a nice way to include the github or pypi path for
    # extension task packages, which could be dynamically installed in a venv at compile
    # time (like pre-commit). https://github.com/moradology/venvception is a nice little
    # package that does this (ephemeral venvs), that a former collaborator wrote for our
    # last project.
  with:
    # kwargs to pass to the task