search

LivePublication / orchestration_logic

Contains logic for exec and monitoring Globus flows using Gladier. Collation and building of orchestration crate.
0 stars 0 forks source link

Rewire Orchestration Logic to provide a workspace for development of the LivePublication CLI #2

Open GusEllerm opened 7 months ago

GusEllerm commented 7 months ago

The Orchestration Logic repo should contain a template-ish instance of how orchestration of a Globus/Gladier flow is integrated with LivePublication. This will be reworked to represent a simple three step pipeline (transfer -> Process (LPAP) -> transfer) which can provide some templating, as well as be a workspace for CLI development.

GusEllerm commented 7 months ago

@NelisDrost The CLI will cover much of the logic within this repo regarding Orchestration crate typing (now provenance crate typing) as well as embedding logic. LidFlow.py is currently specific to the Language Identification case study, and is all about using glaider to execute a flow on Globus. orchestration_crate.py is a class for instantiating the Ocrate and embedding info - this basically needs to all go into the CLI (in a more coherent way).

We need to divide this repo into CLI and 'Globus Orchestration with glacier' components - let me know what you think.

GusEllerm commented 7 months ago

@NelisDrost - here is an example WEP generated using gladier. It should provide enough context to get started with prospective provenance entity embeddings WEP.json. WEP's generated for Globus are written in [https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html](Amazon State's Language) and follows most of the conventions discussed there. Some things such as parallel descriptions dont work with the Globus platform.