Generate Annotated VCF/CSVs for download

davmlaw commented 1 month ago

Downloading a 1-2M record multi-sample VCF with annotations can take so long things time out

I used a few tricks in SACGF/variantgrid_com#86 but we should probably download it then stream a static file

Suggest:

On VCF / Sample page - if there is no CachedGeneratedFile for the url, then display button
Cicking button creates a CachedGeneratedFile + task
This creates CSV/VCF server side behind a UUID
Client side shows message about generation, and has spinner going
Once done, updates CachedGeneratedFile
Drop down a message that says “Creating VCF, download will automatically start if you ” -
Cached generated file check

Current behavior:

view_vcf - cohort_grid_export (csv/vcf) View_sample - sample_grid_export

can probably keep those URLs/

davmlaw commented 1 month ago

Working in branch feature/issue_1171_generate_file_for_download

Mostly done, just need to handle browser bits I think.

We could possibly raise an issue about having analyses handle this as well.

Maybe also have some kind of regular purge of old files?

davmlaw commented 1 month ago

Took a while as I had to do a few things:

Moved CachedGeneratedFile to use UUID so that people couldn't take eg other people's VCFs by brute forcing integer PKs
Needed to zip/gz files as they were very large
Added progress count as files took minutes to generate
Files auto download if you click on page, can come back to page and see progress etc - though won't auto download (will create static link when done, though)

Will probably have to do a few extra things to clean up CachedGeneratedFiles etc over time now as these are quite large still

davmlaw commented 1 month ago

Hmmm, this has taken 2 hours to export 16% - so 8% per hour = 12.5 hours... I think celery is going to time out here...

davmlaw commented 1 month ago

Spun polish scope creep into SACGF/variantgrid#1173 - will do later when less busy

This is VG only, and has been tested and deployed

SACGF / variantgrid