search

sgkit-dev / vcf-zarr-publication

Manuscript and associated scripts for vcf-zarr publication
2 stars 7 forks source link

Add Norway spruce dataset case study #169

Open jeromekelleher opened 16 hours ago

jeromekelleher commented 16 hours ago

@percyfal has applied bio2zarr to a spruce dataset, and it would be an excellent addition to the paper as a further case study.

A notable thing about Norway spruce is the genome size (19Gbp). AFAIK, this means that individual chromosomes overflow the VCF limit of 32 bit integers. As well the usual discussion about compression etc here it would be nice to show that we can overcome these limits by updating the variant_position and contig arrays in place, so that the true genome coordinates are used, and we can show that downstream tools can work on this, unmodified.

This is also a good place to demonstrate the one-big-whole genome Zarr I think?

@percyfal can you update here with details about the dataset, runtimes and the output of vcf2zarr inspect please?

percyfal commented 13 hours ago

Norway spruce has 12 chromosomes totaling some 19Gbp. The assembly consists of more contigs, but I only focus on the main chromosome-level autosomes here. For reasons related to the overflow Jerome mentioned, the VCF files have been generated in 100Mbp chunks over each chromosome, such that there are 160+ files in total. It would indeed be beneficial if we could work in real chromosome coordinates instead of as now having convert between different systems.

I attach the output of vcf2zarr inspect for one test run for both ICF and VCZ. I will provide more information on runtimes when I process the data. Suffice to say that using 128 cores (256 threads) it took only a couple of minutes to convert a 50GiB VCF to ICF and VCZ.

PA_chr01_1.vcz.inspect.txt PA_chr01_1.icf.inspect.txt

jeromekelleher commented 13 hours ago

That's fantastic @percyfal! Could we try doing all 160 files at once?? That might need distributing over the cluster, or if you're happy to wait, a few days on your big machine.

percyfal commented 13 hours ago

I'll see if I can get this going before weekend, the bottleneck is more likely the conversion of gzipped VCFs to bgzip.

percyfal commented 13 hours ago

Just so I'm clear: when you say all VCFs in one go, you mean making one ICF store that then is encoded as VCZ?

jeromekelleher commented 10 hours ago

Yeah, so convert all the VCFs to ICF at the same time using bio2zarr explode [VCF1 VCF2...]. If that's too much work at once, we could just use all the VCFs for one chromosome. It's much easier to stitch the coordinates for a chromosome together if we convert to a single Zarr.

There's no rush at all.