Data related tasks for the 23.12 Platform release

[tskir]

• [x] Release schema repository if updates were implemented.
• [x] Release curation repository if updates were implemented.
• [x] Update the evidence parser repository: EFO, schema, curation versions.
• [x] Run evidence parsers pipelines.
• [x] Release the evidence parser repository.
• [x] Check which external evidence was submitted by the end of the submission window.
• [x] Submit PR to platform-input-support if there are any path changes.
• [x] Run and examine pre-ETL metrics.
• [x] Notify about data submission in Slack.

[DSuveges]

There's a pending PR on evidence datasource parsers (essentiality), it is not urgent, there are a few things still missing. I need to prettify the code and add configuration and the snakemake rule. Nothing mayor, but have no capacity. This is not too urgent, as this step doesn't need to be re-run as there was no update since June.

[DSuveges]

I think there's no blocker for running the parsers. As the ETL is expected to be run on the 27th October week, we can kick off.

[tskir]

Released JSON schema repository: https://github.com/opentargets/json_schema/releases/tag/2.5.0.

Released curation repository: https://github.com/opentargets/curation/releases/tag/23.12.

Updated the parsers repository: pinned EFO/schema/curation versions, as well as some other minor trivial updates. Note that I pinned the EFO version to 3.58.0, the one circulated in the release announcement email. Submitted pull request here: https://github.com/opentargets/evidence_datasource_parsers/pull/175.

Generated and uploaded the internal evidence. Now that the validator isn't giving any trouble, all steps took under an hour of wall time and under 15 minutes of my own time. Everything successfully completed on the first try.

[DSuveges]

Wow, that's amazing... Great job!

[tskir]

Released the evidence parsers repository

https://github.com/opentargets/evidence_datasource_parsers/releases/tag/2.3.0.

Checked which external evidence was submitted

Submitted new evidence for this release:

• gs://otar006-reactome/
• gs://otar012-eva/ — both the target-disease evidence and the pharmacogenomics one

Did not submit new evidence for this release:

• gs://otar007-cosmic — last submission was in June
• gs://otar008-chembl/ — last submission was in August
• gs://otar011-uniprot/ — last submission was in April

Submitted pull request to platform-input-support

Because EVA now submits two types of files and have accordingly changed the layout of their Google Cloud bucket, I've submitted a pull request to platform-input-support to use the new path: https://github.com/opentargets/platform-input-support/pull/112.

[tskir]

Submitted pull requests to ot-release-metrics

To avoid using the temporary gist that we had before, I completed work on https://github.com/opentargets/issues/issues/3123 and submitted two pull requests: https://github.com/opentargets/ot-release-metrics/pull/23 and https://github.com/opentargets/ot-release-metrics/pull/24.

Calculated pre-ETL metrics

Using the above code, I calculated and examined pre-ETL metrics. As usual, they are available in the app as 23.12-pre.

Compared with the previous pre-ETL run

• Compared 23.09-pre → 23.12-pre using the app.
• The only significant change is eva +12,177.
• The other datasources all have very small changes in evidence:
  • Positive: reactome +29, orphanet +1, clingen +84, eva_somatic +86, impc +8
  • Negative: genomics_england –10, gene2phenotype –13
• The tiny negative changes are almost certainly because of EFO class changes, and I don't believe there's a reason to investigate that any further.

Notified about data submission in Slack

Submitted the above results to the #23_12-platform-release channel to notify the backend team and unblock running the ETL.

[tskir]

All actions now done, and I'm closing this issue.

@DSuveges Please additionally let me know if we anticipate any actions in this release in relation to PPP. I've never before worked with PPP data, so I don't know if anything has been done, or is being planned, in that regard.

[DSuveges]

Thanks. All PPP related actions are pushed to 2024.