search

capstone-coal / coal-sds

An Apache OODT-powered Science Data System for COAL
Apache License 2.0
2 stars 3 forks source link

Extend workflow manager to accommodate data acquisition and file management #18

Closed lewismc closed 5 years ago

lewismc commented 5 years ago

Post 0.1 release, once we have stable stand alone services for data acquisition, file management and workflow management (running pycoal tasks), our workflow management should be extended to automate data acquisition and file management. This is a huge move towards the creation of an end-to-end science data system processing capability.

lewismc commented 5 years ago

This issue requires us to investigate the following

  Action:
    Id: TriggerPostIngestWorkflow
    Description: Triggers workflow event with the name [ProductType]Ingest
    Phases: [postIngestSuccess]

Essentially what we want to do here is add the action to crawlctl as follows

--actionIds DeleteDataFile TriggerPostIngestWorkflow

We also need to add the workflow manager URL e.g.

--workflowMgrUrl http://localhost:9001

Once we configure the workflow manager, this will send an event and will enable us to kick off a workflow :)

lewismc commented 5 years ago

This is actually working the other way around where the crawler_launcher kicks off the workflow. All of this will be covered in #9