Use google pub/sub queue in Lira

[kbergin]

Update Lira to add DSS notifications to a google pub/sub topic to fix the timeout/duplicate notifications issue. Add a new endpoint to Lira that subscribes to the topic, does the bundle pre-processing to add a hash of the inputs and checks Cromwell for workflows before submitting a new one.

In the future, we can iterate on this so that Lira saves workflow/hash/bundle info to Redis or Firestore to allow for deduplication and faster look-up of what has been run before.

┆Issue is synchronized with this Jira Story

[kbergin]

➤ Saman Ehsan commented:

PR for review: https://github.com/HumanCellAtlas/lira/pull/177 ( https://github.com/HumanCellAtlas/lira/pull/177|smart-link )

[kbergin]

➤ Saman Ehsan commented:

PR that updates falcon:https://github.com/HumanCellAtlas/falcon/pull/39 ( https://github.com/HumanCellAtlas/falcon/pull/39|smart-link )

[kbergin]

➤ Saman Ehsan commented:

PR to update the tests: https://github.com/HumanCellAtlas/secondary-analysis/pull/785 ( https://github.com/HumanCellAtlas/secondary-analysis/pull/785|smart-link )

[kbergin]

➤ Saman Ehsan commented:

For QA, you can look at the stackdriver logs and see if a message added to the topic was received and processed by Lira to submit an on-hold workflow.

[brianraymor]

Per the DCP PM call on August 22, @jkaneria reported:

We have found additional work, we intend to have the full no-op capability released to staging on Aug 28th and Prod on 9/3. Which I believe slips into milestone 3. I will update the tickets accordingly

Please remember to update your issues. I've already modified the Milestone.

[jkaneria]

We will try an out-of-cycle deployment to staging tomorrow to match up to previous plans.

[kbergin]

➤ Chengchen Wang commented:

QAed!