search

TIGRLab / air-tigrs

Orchestration tools for TIGRLab's data management infrastructure
1 stars 5 forks source link

Initial offering of DAG factory for task parsers #21

Open benselby opened 2 years ago

benselby commented 2 years ago

This is an example of how the DAG factory pattern can be used for task parsers. Right now it's quite simplistic: a TaskParser key is added to the study's settings.yml, which lists tasks and the appropriate command. As of now this means it does not support any dynamic arguments to whatever command is being issued - we can discuss if this is an important feature or not.

So for example, SPN20_settings.yml needs to contain the additional key:

TaskParser:
    EA: dm_parse_ea.py SPN20 --debug --regex "*MR*EMP*"

Studies can specify multiple task parsers, e.g. for OPT:

TaskParser:
        FACES: dm_parse_faces.py OPT
        FAKE: echo "FAKE OUT!"

I still need to write tests for this implementation but thought I would post for feedback beforehand since it's been a long time coming!