Seqr Cost Recovery

[illusional]

As part of recovering costs for external datasets, we need to store and determine some reporting mechanism for the different costs we accrue for processing and storing data (re @lgruen):

• CRAM storage cost (should be straightforward if we have dedicated datasets / GCP projects)
• Alignment + variant calling (more difficult, but hopefully can be integrated with per-job Hail costs stored as part of the sample metadata?)
• Joint calling and annotation (somewhat difficult; the best idea I can come up with is to prorate this based on the relative CRAM sizes; we’ve discussed using the number of variants as another proxy before, but that seems even trickier given that many variants will be shared across the datasets)
• Deployment costs (Elasticsearch, Postgres, VM to run Django: suggesting again to prorate this based on the amount of data, e.g. based on CRAMs)

As part of doing this, we need to ensure that costs are counted in the correct currency, and for the correct region.

[illusional]

@vladsaveliev, I've assigned us in the short term, but mostly just to keep us thinking about it. I'm having a deeper look into hail batch billing costs at the moment.

[illusional]

Noting here that the hail batch costs are derived from the resource table, which by default stores cost for the us-east1 region (eg: populationgenomics/hail:batch/sql/insert_nonpreemptible_resources.py#L13-L14).

There are a couple of things we'll need to consider:

• How to handle choosing the correct rate (do we rewrite the values in the table, support multiple region and link them back later).
• When to decide the USD to AUD conversion
• Cost difference between GCP and Azure.
• Displaying the cost to the user on the UI.

[violetbrina]

Planning document

[violetbrina]

• Query: Per topic cost per invoice month, include sum of cram sizes
• Query: Per sample per month cost of everything non-seqr (service.id != seqr) -> data loading cost

Write both into a python script that dumps out a csv for loading elsewhere.