cannot join g.vcf files using glnexus

[kostasgalexiou]

Hi,

When I try to merge gvcf files generated with deepvariant, using glnexus, I get the following error:

[W::bgzf_read_block] EOF marker is absent. The input is probably truncated Error: BCF read error

Below is the command that I am using:

./glnexus_cli \ --config DeepVariantWGS \ SQU1.sort.pcr_rem.RG.kept.g.vcf.gz SQU3.sort.pcr_rem.RG.kept.g.vcf.gz | \ bcftools view - | \ bgzip -c > SQU13.vcf.gz

My gvcf files look normal and the log file for gvcf generation did not give any errors.

Any help will be appreciated.

Thanks

[mlin]

The mentioned "EOF marker" consists of the following 28 hex bytes. Can you try tail -c 28 SQU1.sort.pcr_rem.RG.kept.g.vcf.gz | xxd -p (and your other file too) to confirm if it's present or not:

1f8b08040000000000ff0600424302001b0003000000000000000000

This will just adjudicate whether the message displayed is true or not.

[mlin]

BTW, please also check the log for other error messages preceding that one. It's possible the one you showed actually comes from your bcftools view - command complaining because GLnexus' output is truncated due to some other error.

[kostasgalexiou]

The mentioned "EOF marker" consists of the following 28 hex bytes. Can you try tail -c 28 SQU1.sort.pcr_rem.RG.kept.g.vcf.gz | xxd -p (and your other file too) to confirm if it's present or not:

1f8b08040000000000ff0600424302001b0003000000000000000000

This will just adjudicate whether the message displayed is true or not.

Dear @mlin,

Thanks for you fasta reply. I have checked all my gvcf files and all of them contain the EOF marker

[mlin]

Please show more of the log leading up to the error message

[kostasgalexiou]

BTW, please also check the log for other error messages preceding that one. It's possible the one you showed actually comes from your bcftools view - command complaining because GLnexus' output is truncated due to some other error.

[kostasgalexiou]

Please show more of the log leading up to the error message

./glnexus_cli --config DeepVariantWGS SQU1.sort.pcr_rem.RG.kept.g.vcf.gz SQU3.sort.pcr_rem.RG.kept.g.vcf.gz | bcftools view - | bgzip -c > SQU13.vcf.gz [29743] [2021-07-03 17:58:32.006] [GLnexus] [info] glnexus_cli release v1.2.6-0-g4d057dc Wed May 20 22:14:49 2020 [29743] [2021-07-03 17:58:32.006] [GLnexus] [warning] jemalloc absent, which will impede performance with high thread counts. See https://github.com/dnanexus-rnd/GLnexus/wiki/Performance [29743] [2021-07-03 17:58:32.006] [GLnexus] [info] Loading config preset DeepVariantWGS [29743] [2021-07-03 17:58:32.011] [GLnexus] [info] config: unifier_config: drop_filtered: false min_allele_copy_number: 1 min_AQ1: 10 min_AQ2: 10 min_GQ: 0 max_alleles_per_site: 32 monoallelic_sites_for_lost_alleles: true preference: common genotyper_config: revise_genotypes: true min_assumed_allele_frequency: 9.99999975e-05 required_dp: 0 allow_partial_data: true allele_dp_format: AD ref_dp_format: MIN_DP output_residuals: false more_PL: true squeeze: false trim_uncalled_alleles: true output_format: BCF liftover_fields:

• {orig_names: [MIN_DP, DP], name: DP, description: "##FORMAT=<ID=DP,Number=1,Type=Integer,Description=\"Approximate read depth (reads with MQ=255 or with bad mates are filtered)\">", type: int, number: basic, default_type: missing, count: 1, combi_method: min, ignore_non_variants: true}
• {orig_names: [AD], name: AD, description: "##FORMAT=<ID=AD,Number=R,Type=Integer,Description=\"Allelic depths for the ref and alt alleles in the order listed\">", type: int, number: alleles, default_type: zero, count: 0, combi_method: min, ignore_non_variants: false}
• {orig_names: [GQ], name: GQ, description: "##FORMAT=<ID=GQ,Number=1,Type=Integer,Description=\"Genotype Quality\">", type: int, number: basic, default_type: missing, count: 1, combi_method: min, ignore_non_variants: true}
• {orig_names: [PL], name: PL, description: "##FORMAT=<ID=PL,Number=G,Type=Integer,Description=\"Phred-scaled genotype Likelihoods\">", type: int, number: genotype, default_type: missing, count: 0, combi_method: missing, ignore_non_variants: true} [29743] [2021-07-03 17:58:32.011] [GLnexus] [info] config CRC32C = 2857227159 [29743] [2021-07-03 17:58:32.011] [GLnexus] [info] init database, exemplar_vcf=SQU1.sort.pcr_rem.RG.kept.g.vcf.gz [29743] [2021-07-03 17:58:33.793] [GLnexus] [info] Initialized GLnexus database in GLnexus.DB [29743] [2021-07-03 17:58:33.793] [GLnexus] [info] bucket size: 30000 [29743] [2021-07-03 17:58:33.793] [GLnexus] [info] contigs: Cp4.1LG01 Cp4.1LG02 Cp4.1LG03 Cp4.1LG04 Cp4.1LG05 Cp4.1LG06 Cp4.1LG07 Cp4.1LG08 Cp4.1LG09 Cp4.1LG10 Cp4.1LG11 Cp4.1LG12 Cp4.1LG13 Cp4.1LG14 Cp4.1LG15 Cp4.1LG16 Cp4.1LG17 Cp4.1LG18 Cp4.1LG19 Cp4.1LG20 Cp4.1LG00 [29743] [2021-07-03 17:58:35.067] [GLnexus] [info] db_get_contigs GLnexus.DB [29743] [2021-07-03 17:58:35.409] [GLnexus] [info] Beginning bulk load with no range filter. [29743] [2021-07-03 18:00:21.063] [GLnexus] [info] Loaded 2 datasets with 2 samples; 9525882024 bytes in 105699217 BCF records (12780 duplicate) in 17576 buckets. Bucket max 863424 bytes, 9394 records. 0 BCF records skipped due to caller-specific exceptions [29743] [2021-07-03 18:00:21.064] [GLnexus] [info] Created sample set @2 [29743] [2021-07-03 18:00:21.064] [GLnexus] [info] Flushing database... [29743] [2021-07-03 18:00:25.980] [GLnexus] [info] Bulk load complete! [29743] [2021-07-03 18:00:25.983] [GLnexus] [warning] Processing full length of 21 contigs, as no --bed was provided. Providing a BED file with regions of interest, if applicable, can speed this up. [29743] [2021-07-03 18:00:25.986] [GLnexus] [info] found sample set @2 [29743] [2021-07-03 18:00:25.986] [GLnexus] [info] discovering alleles in 21 range(s) on 6 threads [29743] [2021-07-03 18:01:08.085] [GLnexus] [info] discovered 2858339 alleles [29743] [2021-07-03 18:01:11.026] [GLnexus] [info] unified to 1254278 sites cleanly with 1261642 ALT alleles. 527 ALT alleles were additionally included in monoallelic sites and 171215 were filtered out on quality thresholds. [29743] [2021-07-03 18:01:11.026] [GLnexus] [info] Finishing database compaction... [29743] [2021-07-03 18:01:11.101] [GLnexus] [info] genotyping 1254278 sites; sample set = *@2 mem_budget = 0 threads = 8 [W::bgzf_read_block] EOF marker is absent. The input is probably truncated Error: BCF read error

[kostasgalexiou]

Hi @mlin,

Sorry for closing the issue. Is still remains open.

Regards

[mlin]

Please try writing the glnexus_cli output to disk (without the subsequent shell pipeline) just to see what happens then or if there's a different error message. Something appears to be stopping it prematurely in the genotyping stage (which comes after it will have read in the complete GVCF inputs -- I'm fairly certain the "EOF marker is absent" is coming from bcftools in the pipeline). The most common cause of that is the OS out-of-memory killer, but there's typically a message like "Killed" in that case so I'm not sure if that's the case here.

[kostasgalexiou]

Hi @mlin,

Looks like it is a memory failure...

[16357] [2021-07-05 09:34:42.832] [GLnexus] [info] glnexus_cli release v1.2.6-0-g4d057dc Wed May 20 22:14:49 2020 [16357] [2021-07-05 09:34:42.832] [GLnexus] [warning] jemalloc absent, which will impede performance with high thread counts. See https://github.com/dnanexus-rnd/GLnexus/wiki/Performance [16357] [2021-07-05 09:34:42.832] [GLnexus] [info] Loading config preset DeepVariantWGS [16357] [2021-07-05 09:34:43.050] [GLnexus] [info] config: unifier_config: drop_filtered: false min_allele_copy_number: 1 min_AQ1: 10 min_AQ2: 10 min_GQ: 0 max_alleles_per_site: 32 monoallelic_sites_for_lost_alleles: true preference: common genotyper_config: revise_genotypes: true min_assumed_allele_frequency: 9.99999975e-05 required_dp: 0 allow_partial_data: true allele_dp_format: AD ref_dp_format: MIN_DP output_residuals: false more_PL: true squeeze: false trim_uncalled_alleles: true output_format: BCF liftover_fields:

• {orig_names: [MIN_DP, DP], name: DP, description: "##FORMAT=<ID=DP,Number=1,Type=Integer,Description=\"Approximate read depth (reads with MQ=255 or with bad mates are filtered)\">", type: int, number: basic, default_type: missing, count: 1, combi_method: min, ignore_non_variants: true}
• {orig_names: [AD], name: AD, description: "##FORMAT=<ID=AD,Number=R,Type=Integer,Description=\"Allelic depths for the ref and alt alleles in the order listed\">", type: int, number: alleles, default_type: zero, count: 0, combi_method: min, ignore_non_variants: false}
• {orig_names: [GQ], name: GQ, description: "##FORMAT=<ID=GQ,Number=1,Type=Integer,Description=\"Genotype Quality\">", type: int, number: basic, default_type: missing, count: 1, combi_method: min, ignore_non_variants: true}
• {orig_names: [PL], name: PL, description: "##FORMAT=<ID=PL,Number=G,Type=Integer,Description=\"Phred-scaled genotype Likelihoods\">", type: int, number: genotype, default_type: missing, count: 0, combi_method: missing, ignore_non_variants: true} [16357] [2021-07-05 09:34:43.050] [GLnexus] [info] config CRC32C = 2857227159 [16357] [2021-07-05 09:34:43.050] [GLnexus] [info] init database, exemplar_vcf=SQU1.sort.pcr_rem.RG.kept.g.vcf.gz [16357] [2021-07-05 09:34:44.833] [GLnexus] [info] Initialized GLnexus database in GLnexus.DB [16357] [2021-07-05 09:34:44.833] [GLnexus] [info] bucket size: 30000 [16357] [2021-07-05 09:34:44.833] [GLnexus] [info] contigs: Cp4.1LG01 Cp4.1LG02 Cp4.1LG03 Cp4.1LG04 Cp4.1LG05 Cp4.1LG06 Cp4.1LG07 Cp4.1LG08 Cp4.1LG09 Cp4.1LG10 Cp4.1LG11 Cp4.1LG12 Cp4.1LG13 Cp4.1LG14 Cp4.1LG15 Cp4.1LG16 Cp4.1LG17 Cp4.1LG18 Cp4.1LG19 Cp4.1LG20 Cp4.1LG00 [16357] [2021-07-05 09:34:45.221] [GLnexus] [info] db_get_contigs GLnexus.DB [16357] [2021-07-05 09:34:45.637] [GLnexus] [info] Beginning bulk load with no range filter. [16357] [2021-07-05 09:36:55.487] [GLnexus] [info] Loaded 2 datasets with 2 samples; 9525882024 bytes in 105699217 BCF records (12780 duplicate) in 17576 buckets. Bucket max 863424 bytes, 9394 records. 0 BCF records skipped due to caller-specific exceptions [16357] [2021-07-05 09:36:55.487] [GLnexus] [info] Created sample set @2 [16357] [2021-07-05 09:36:55.487] [GLnexus] [info] Flushing database... [16357] [2021-07-05 09:37:00.598] [GLnexus] [info] Bulk load complete! [16357] [2021-07-05 09:37:00.602] [GLnexus] [warning] Processing full length of 21 contigs, as no --bed was provided. Providing a BED file with regions of interest, if applicable, can speed this up. [16357] [2021-07-05 09:37:00.605] [GLnexus] [info] found sample set @2 [16357] [2021-07-05 09:37:00.605] [GLnexus] [info] discovering alleles in 21 range(s) on 6 threads [16357] [2021-07-05 09:37:47.103] [GLnexus] [info] discovered 2858339 alleles [16357] [2021-07-05 09:37:50.245] [GLnexus] [info] unified to 1254278 sites cleanly with 1261642 ALT alleles. 527 ALT alleles were additionally included in monoallelic sites and 171215 were filtered out on quality thresholds. [16357] [2021-07-05 09:37:50.245] [GLnexus] [info] Finishing database compaction... [16357] [2021-07-05 09:37:50.321] [GLnexus] [info] genotyping 1254278 sites; sample set = *@2 mem_budget = 0 threads = 8 Terminado (killed)

[kostasgalexiou]

I suppose I can run the analysis per chromosome.

[mlin]

Try setting the command-line option --mem-gbytes 1 or something else conservative. It's a trade-off where (at scale) it will be faster with more memory at its disposal, short (obviously) of getting killed.

[kostasgalexiou]

It has worked with the --mem-gbytes 1 argument! Thanks a lot for the help and your prompt replies!