Generate Annotated VCF/CSVs for download

[davmlaw]

Downloading a 1-2M record multi-sample VCF with annotations can take so long things time out

I used a few tricks in SACGF/variantgrid_com#86 but we should probably download it then stream a static file

Suggest:

• On VCF / Sample page - if there is no CachedGeneratedFile for the url, then display button
• Cicking button creates a CachedGeneratedFile + task
• This creates CSV/VCF server side behind a UUID
• Client side shows message about generation, and has spinner going
• Once done, updates CachedGeneratedFile
• Drop down a message that says “Creating VCF, download will automatically start if you ” -
• Cached generated file check

Current behavior:

view_vcf - cohort_grid_export (csv/vcf) View_sample - sample_grid_export

can probably keep those URLs/

[davmlaw]

Working in branch feature/issue_1171_generate_file_for_download

Mostly done, just need to handle browser bits I think.

We could possibly raise an issue about having analyses handle this as well.

Maybe also have some kind of regular purge of old files?

[davmlaw]

Took a while as I had to do a few things:

• Moved CachedGeneratedFile to use UUID so that people couldn't take eg other people's VCFs by brute forcing integer PKs
• Needed to zip/gz files as they were very large
• Added progress count as files took minutes to generate
• Files auto download if you click on page, can come back to page and see progress etc - though won't auto download (will create static link when done, though)

Will probably have to do a few extra things to clean up CachedGeneratedFiles etc over time now as these are quite large still

[davmlaw]

Hmmm, this has taken 2 hours to export 16% - so 8% per hour = 12.5 hours... I think celery is going to time out here...

[davmlaw]

Spun polish scope creep into SACGF/variantgrid#1173 - will do later when less busy

This is VG only, and has been tested and deployed

[davmlaw]

Rework: ended up not waiting for node properly

Fixed, need vg test