Investigate how many projects have an exact match ENA/HCA filenames

[ESapenaVentura]

Background: HCA download costs from the HCA Data Portal are rising. We need to find a way to lower costs. One option would be to offer download from the EBI archives for datasets whose files are available there.

Current findings:

• We have a total of 439301 files (sequence + analysis + supplementary)
• We have a total of 390269 files with accession recorded in the metadata (content.insdc_run_accessions filled).
• From the total, only 8716 files are analysis files and 60 are supplementary files, so that leaves 40256 files that are potentially not in archives (or that need assessment in order to determine if they are really not in public archives)

Questions:

• [x] How many of these 390269 files correspond to published projects in the Data portal?
• [x] Create a map file --> project for the ones above
• [x] How many of these files have a exact HCA filename / SRA-ENA filename mapping?

[ESapenaVentura]

Investigating the third question. I have written a script with certain optimisations:

• Read HCA filenames from mongoDB backup
• Assume if HCA filename startswith SRR or ERR, can be retrieved from ENA, without API calls to confirm

Even so, it estimates around 88 hours (~4 days) of uninterrupted running to finish.

If you can think of any other optimisation/assumption let me know!

[ESapenaVentura]

There is a raw report with supporting files here

We're going to meet with tony to discuss about it. Until then, I'm not performing additional analysis

[gabsie]

Awesome work, @ESapenaVentura We have enough for the conversation with JR this week, and we will take it from there.