Design: Compiling parameters for Resuming and re-running jobs POC

icgc-argo / workflow-graph-node

Workflow Graph Node

GNU Affero General Public License v3.0

0 stars 0 forks source link

Design: Compiling parameters for Resuming and re-running jobs POC #73

Closed akachru-github closed 2 years ago

akachru-github commented 3 years ago

Currently, Model T talks hardcoded parameters for each type of workflow; reads from sheet and replaces the dynamic values, then sends the run with resume = true

Resume: Same session id; can have the same or different parameters. Must have the same working directory to access interstitial files. Rerun: New session id; totally fresh. can have the same or different parameters. Can have a new working directory.

Workflow automation does not have an equivalent yet, so we need to make these functions available.

Possible Implementation:

hosting static parameters somewhere? Maybe as an optional default field in AVRO schema.
Resume endpoint with partial parameters provided + runid in node api
It completes a merge of old parameters + the provided parameters. If no parameters are provided, it uses all the same parameters with resume = true.

Expected Outcome

Wiki + next steps tickets!

Edit: New wiki with design has been created here

lepsalex commented 3 years ago

Turns out that AVRO default values have nothing to do with read/write of schema-based object and only to do with schema migration ...

default: A default value for this field, only used when reading instances that lack the field for schema evolution 
purposes. The presence of a default value does not make the field optional at encoding time. Permitted values
depend on the field's schema type, according to the table below. Default values for union fields correspond to
the first schema in the union. Default values for bytes and fixed fields are JSON strings, where Unicode code points
0-255 are mapped to unsigned 8-bit byte values 0-255. Avro encodes a field even if its value is equal to its default.

Docs: https://avro.apache.org/docs/current/spec.html

Further clarified here: https://stackoverflow.com/questions/58880481/why-is-the-avro-default-value-not-used-with-avro-python

The search for the right solution continues ...

rosibaj commented 3 years ago

@rosibaj scheduled a meeting Oct 1 to confirm business requirements

do we TRS?
do we build a new service?

akachru-github commented 3 years ago

May require collaboration with GA4GH (Jeremy) and Dockstore (Dennis). Need someone to follow through with this as this is a large and complex change. The changes being requested aren't ARGO specific and can be changes for general purpose use.

akachru-github commented 3 years ago

There is still work to do on fleshing this out further.

jaserud commented 2 years ago

Closing this since it was design which was done and wiki created. Implementation continues here: https://github.com/icgc-argo/workflow-graph-node/issues/85