search

capstone-coal / coal-sds

An Apache OODT-powered Science Data System for COAL
Apache License 2.0
2 stars 3 forks source link

Implement Workflows in COAL-SDS #9

Open lewismc opened 6 years ago

lewismc commented 6 years ago

For full end-to-end automation, we need to eventually implement OODT's Workflow Management capabilities.

lewismc commented 5 years ago

@alexwyi19 @ahuerta33 @DenimMazuki @kristencheung can you please assign yourselves to whatever issue we had previous discussed? Thanks

lewismc commented 5 years ago 
./crawler_launcher --printSupportedCrawlerActions

https://github.com/capstone-coal/coal-sds/wiki/Automation-of-Data-Acquisition-and-Crawling Flags to add are 

-wm,    --workflowMgrUrl <url> which is most likely http://localhost:9001

Additionally you need to add a post ingestion actionId

e.g. 
  Action:
    Id: TriggerPostIngestWorkflow
    Description: Triggers workflow event with the name [ProductType]Ingest
    Phases: [postIngestSuccess]

This should appended to 

--actionIds DeleteDataFile
lewismc commented 5 years ago

The workflow documentation for OODT resides at http://oodt.apache.org/site_docs/cas-workflow/ you can see the bottom left=hand-side for the navigation links to http://oodt.apache.org/site_docs/cas-workflow/development/developer.html http://oodt.apache.org/site_docs/cas-workflow/user/basic.html http://oodt.apache.org/site_docs/cas-workflow/user/advanced.html I would read them through thoroughly and then begin working within the workflow component for coal-sds.

lewismc commented 5 years ago

I've committed a lot of code to cover this issue e.g. https://github.com/capstone-coal/coal-sds/commit/fc1418de9085e397ff6b4fe1a474d3b09801cfa8 There is more work to do be to automate the initiation of the workflows directly after the crawler_launcher has successfully ingested the AVIRIS products. Additionally, the new workflow policy needs to be documented such that workflows are easily understood. This will however be covered in https://github.com/capstone-coal/coal-sds/issues/13