POC: Define a Pipeline to test workflow-graph services

[rosibaj]

Since the Node/Edge Services are completed we need to test the viability of the pipeline connections. We wil hardcoded a pipeline and have it use all the services to verify that each piece is working as expected.

• configure ingest node a Song publish event for sequencing _experiment
• configure a workflow node for alignment
• configure a workflow node for sanger wxs (logic: for the donor, T/N alignment analysis must both exist, and alignment is wxs)

Expected Outcome

Publish a sequencing_experiment analysis

• see normal alignment kick off (seen in wes ui)
• see tumour alignment kick off (seen in wes ui)
• see sanger kick off (seen in wes ui)

Test Data

Dev (same files and meta info as in qa just different analysis/study/donor/sample/specimen ids)

WXS Tumour BAM analysis: https://song.rdpc-dev.cancercollaboratory.org/studies/RDPCDEV-CA/analysis/aee88104-1d07-447d-a881-041d07647ddb

WXS Normal BAM analysis: https://song.rdpc-dev.cancercollaboratory.org/studies/RDPCDEV-CA/analysis/519f3db9-e34f-44eb-9f3d-b9e34f34eb62

Qa

WXS Tumour BAM analysis: https://song.rdpc-qa.cancercollaboratory.org/studies/TEST-PR/analysis/1d891219-5b8e-4bc3-8912-195b8ebbc3d7 alignment running duration: ~15mins

WXS Normal BAM analysis: https://song.rdpc-qa.cancercollaboratory.org/studies/TEST-PR/analysis/56847fc7-7457-4df2-847f-c77457edf2a4 alignment running duration: ~12mins

Sanger-WXS running duration: ~10mins

Note: Exploration during this POC

• think about how to enter the pipeline at individual nodes (not the start node)
• think about how to assess whether different workflow nodes need to be run in a newly versioned pipeline

[rosibaj]

@lindaxiang let's connect on a test data set here.

[lindaxiang]

@rosibaj sure thing. let me find the proper test data set for you.

[rosibaj]

@lindaxiang Please indicate a:

• Whole Exome Tumour BAM analysis_id
• Whole Exome Normal BAM analysis_id

[lindaxiang]

• WXS Tumour BAM analysis: https://song.rdpc-qa.cancercollaboratory.org/studies/TEST-PR/analysis/1d891219-5b8e-4bc3-8912-195b8ebbc3d7 alignment running duration: ~15mins

• WXS Normal BAM analysis: https://song.rdpc-qa.cancercollaboratory.org/studies/TEST-PR/analysis/56847fc7-7457-4df2-847f-c77457edf2a4 alignment running duration: ~12mins

• Sanger-WXS running duration: ~10mins

[rosibaj]

@rosibaj Need to get this test data into dev after namespace work - @yalturmes can you let me know when dev is back after the namespace/empty-dir work is done?

[andricDu]

Progress Update

@lepsalex @rosibaj My opinion is that there is just a little bit more refinement of the components needed before the solution works end to end. I've documented my progress and findings in this update.

Deployment

Ingest Node (RDPC-QA)

Deployed: https://github.com/icgc-argo/workflow-roadmap/issues/60

$ k get deployment ingest-node
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
ingest-node   1/1     1            1           6d23h

Alignment Node (RDPC-QA)

Deployed: https://github.com/icgc-argo/workflow-roadmap/issues/61 Running workflows: https://github.com/icgc-argo/dna-seq-processing-wfs.git with revision: 1.5.1

$ k get deployment align-node 
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
align-node   1/1     1            1           80m

Example of launched workflow: https://wes.rdpc-qa.cancercollaboratory.org/runs/wes-f02af600905540ed99a24cb228499f87

Sanger VC Node

No progress.

Development work required

Several issues were identified that are blockers on having the system working end to end.

• Ingest node will halt execution when encountering an error. https://github.com/icgc-argo/workflow-graph-ingest-node/issues/11
• Song Kafka and Ingest Node are currently incompatible. There will need to be a refactor to fix this. https://github.com/icgc-argo/workflow-graph-ingest-node/issues/12
  • blocked by / waiting on: https://github.com/overture-stack/SONG/issues/674
• RDPC Client is not correctly handling certain exceptions causing them to be dlqed rather than retried: https://github.com/icgc-argo/workflow-graph-lib/issues/27
• There appears to be silent DLQing of messages on the boundary between Ingest Node, and Workflow Graph Node. What appear to be valid GraphEvents are being DLQed in the input dlq. No logs show up to indicate why https://github.com/icgc-argo/workflow-graph-node/issues/41

Extra Notes

• Input Params Schema for Testing alignemnt has been registered:

  
  {
  "type" : "record",
  "name" : "AlignParamsTest",
  "namespace": "org.icgc_argo.workflow_graph_lib.schema",
  "fields" : [
    {
      "name" : "analysis_id" ,
      "type" : "string"
    },
    {
      "name" : "study_id" ,
      "type" : "string"
    },
    {
      "name" : "score_url" ,
      "type" : "string"
    },
    {
      "name" : "song_url" ,
      "type" : "string"
    },
    {
      "name": "ref_genome_fa",
      "type": "string"
    },
    {
      "name": "download",
      "type": {
          "type": "record",
          "name": "downloadRecord",
          "fields" : [
              {
                  "name" : "song_cpus" ,
                  "type" : "int"
              },
              {
                  "name" : "song_mem" ,
                  "type" : "int"
              },
              {
                  "name" : "score_cpus" ,
                  "type" : "int"
              },
              {
                  "name" : "score_mem" ,
                  "type" : "int"
              },
              {
                  "name": "score_url",
                  "type": "string"
              },
              {
                  "name": "song_url",
                  "type": "string"
              }
          ]
      }
    },
    {
        "name": "cpu",
        "type": "int"
    },
    {
        "name": "mem",
        "type": "int"
    }
  ]
  }

[andricDu]

I'm using the following: https://github.com/icgc-argo/argo-sandbox/tree/master/wfg as scratch space for my work with the deployment.

[akachru-github]

All work to support this is now in QA.