Feature Request: Dev and QA Integration tests - Download

[rosibaj]

Score was deployed with a bad value, resulting in errors when attempting to use both the dev and qa environments (https://github.com/overture-stack/score/issues/290). The errors were not entirely clear to the end user (Linda + Rosi).

Dusan discovered that the Score server needs to use --reuse-values when being deployed, and it was not, leading to a bad deployment configuration.

We need to prevent deployment mistakes like this before they happen.

Implementation

Set up a download integration test. For a file that exists in that environment, download via docker/score-client.

• similar to ICGC Download (runtime etc)

• [ ] Dev

• [x] QA

• [x] Prod

[akachru-github]

for ICGC

• We benchmark the download speed
• Sort of an e2e test
• The question we wanted to answer here: is our Ceph cluster providing a decent throughput?

Suggestion:

• We do not need benchmarks for ARGO
• Use a small group of smaller files that are randomly selected every hour. They don't need to be huge.
• This is a functional test.
• Since both ICGC and ARGO are backed by the same infra, performance measurements in both places would be redundant
• Reporting: Graph in grafana
• Failures will result in alerts

[jmimico]

I have set up this automated download test for ARGO data however the ARGO token is only live for ~30 days. This is not acceptable for an automated test. ICGC legacy allows for a token that lives for 365 days. @akachru-github I consider this token a bit of a blocker, I don't want to be manually updating tokens every 30 days.

[akachru-github]

@jmimico this is agains the prod env?

[jmimico]

Yes, I am setting this up for prod first just because everything in prod is well documented, stable, and live. Don't know anything about dev/qa yet @akachru-github

[jmimico]

@rosibaj and @andricDu have coached me through the process of a registered application in ego and how to retrieve the token. I tested this and it works so I am unblocked!

Need to integrate this token generation piece in to the download script now.

[jmimico]

I have set up the automated download tests for PROD and QA.

PROD picks from a list of CRAM's that range from 2.0-2.9GB file every hour on the half hour. QA downloads a single static object every hour on the :35.

These downloads are graphed: PROD: https://monitor.cancercollaboratory.org:3001/d/eRMLtQsWz/argo-rdpc-operations?orgId=1 QA: https://monitor.cancercollaboratory.org:3001/d/knQpV3oMk/argo-rdpc-dev-qa-operations?orgId=1

If these downloads fail and are not charting data for 4 hours, they will alert (from Grafana) to #argo-rdpc-alerts and #argo-rdpc-dev-qa-alerts respectively as well as #zabbix_alerts .