Potential problem with writing gz files

[jrm5100]

I'm using fgoxide to write a test involving .vcf.gz files.

I've defined some input lines like

let tempdir = TempDir::new().unwrap();
let io = fgoxide::io::Io::default();

let input = tempdir.path().join("input.vcf.gz");
let input_data = vec![
        r#"##fileformat=VCFv4.0 "#,
        r#"##build_id=20201027095038"#,
        r#"##Population=https://www.ncbi.nlm.nih.gov/biosample/?term=GRAF-pop"#,
        r#"##FORMAT=<ID=AN,Number=1,Type=Integer,Description="Total allele count for the population, including REF">"#,
        r#"##FORMAT=<ID=AC,Number=A,Type=Integer,Description="Allele count for each ALT allele for the population">"#,
        r#"#CHROM   POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  SAMN10492695    SAMN10492696    SAMN10492697    SAMN10492698    SAMN10492699    SAMN10492700    SAMN10492701    SAMN10492702    SAMN11605645    SAMN10492703    SAMN10492704    SAMN10492705"#,
        r#"NC_000001.9  144135212   rs1553120241    G   A   .   .   .   AN:AC   8560:5387   8:8 256:224 336:288 32:24   170:117 32:24   18:13   20:15   344:296 288:248 9432:6100"#,
        r#"NC_000001.9  144148243   rs2236566   G   T   .   .   .   AN:AC   5996:510    0:0 0:0 0:0 0:0 0:0 0:0 0:0 84:8    0:0 0:0 6080:518"#,
] ;

io.write_lines(&input, &input_data)?;

My test is failing with Error: bytes remaining on stream, but passing with files I've run manually.

If I extract the generated VCF file and re-compress it with bgzip

gunzip input.vcf.gz
bgzip -c input.vcf > input2.vcf.gz

My tool works. The same issue occurs when running the tool directly on the files (so the test itself isn't the problem, unless I've written the input file incorrectly).

The bgzip version is slightly larger.

There's no difference in the vcf version of each.

diff <(gzcat test_path/input.vcf.gz) <(gzcat test_path/input2.vcf.gz)

input.vcf.gz input2.vcf.gz

[tfenne]

@jrm5100 I wonder - how is your tool reading the VCF? I'm fairly sure that io::write_lines() will write a vanilla gzipped file if the extension is .gz - not a bgzipped file. Perhaps the tool you're running expects gzipped files to be bgzipped?

If that is the case I'd also be open to making io::write_lines() write bgzipped instead of plain gzipped files as that's probably more generally useful.

[jrm5100]

Yeah, it just occurred to me the incompatibility might be with noodles-bgzf, and I guess the name might confirm that your hunch is correct.

That's what I get for trying to write up a strange issue on a Friday evening without thinking it through.

Not sure what the best approach is. Perhaps .vcf.gz should write bgzip files (since that is the norm for vcf), but I don't know that I'd want to encode too much logic into how file extensions are written.

[nh13]

What about the following simple rust psuedo-code that could be refactored into its own function. If it's only for testing purposes, that should work. We tend to use this library for compression: https://github.com/sstadick/gzp

let tempdir = TempDir::new().unwrap();
let io = fgoxide::io::Io::default();
let input = tempdir.path().join("input.vcf.gz");
let writer = BufWriter::new(File::create(&input).unwrap());
let mut bgzf_writer = BgzfSyncWriter::new(writer, Compression::new(3));
bgzf_writer.write_all(b"@NAME\nGATTACA\n+\nIIIIIII\n").unwrap();
bgzf_writer.flush().unwrap();
drop(gz_writer);