ERROR WHILE RUNNING TBPROFILER: Kindly help me solve this error

[SandraBabirye]

tb-profiler error report

• OS: linux
• Program version: 4.4.2
• Database version: c2fb9a2
• Program call:

  {'read1': '/etc/ace-data/home/sbabirye/data/fastq_Seq/ERR038736_LIB1_R1.fastq.gz', 'read2': '/etc/ace-data/home/sbabirye/data/fastq_Seq/ERR038736_LIB2_R2.fastq.gz', 'bam': None, 'fasta': None, 'vcf': None, 'platform': 'illumina', 'db': 'tbdb', 'external_db': None, 'prefix': 'ERR038736', 'csv': False, 'txt': True, 'pdf': False, 'docx': None, 'output_template': None, 'add_columns': None, 'add_mutation_metadata': False, 'dir': '.', 'mapper': 'bwa', 'caller': 'freebayes', 'calling_params': None, 'kmer_counter': 'kmc', 'min_depth': 10, 'af': 0.1, 'reporting_af': 0.1, 'coverage_fraction_threshold': 0, 'missing_cov_threshold': 10, 'suspect': False, 'spoligotype': False, 'call_whole_genome': False, 'snp_dist': None, 'no_trim': False, 'no_flagstat': False, 'no_clip': True, 'no_delly': False, 'no_lineage': False, 'add_variant_annotations': False, 'threads': 4, 'ram': 2, 'verbose': 0, 'delly_vcf': None, 'no_clean': False, 'temp': '.', 'func': <function main_profile at 0x7fcd14e5b910>, 'software_name': 'tbprofiler', 'tmp_prefix': '60ef5aaf-1eb5-4bb7-9ed7-8e628b7f249c', 'files_prefix': './60ef5aaf-1eb5-4bb7-9ed7-8e628b7f249c', 'conf': {'snpEff_db': 'Mycobacterium_tuberculosis_h37rv', 'drugs': ['rifampicin', 'isoniazid', 'ethambutol', 'pyrazinamide', 'streptomycin', 'fluoroquinolones', 'moxifloxacin', 'ofloxacin', 'levofloxacin', 'ciprofloxacin', 'aminoglycosides', 'amikacin', 'capreomycin', 'kanamycin', 'cycloserine', 'ethionamide', 'clofazimine', 'para-aminosalicylic_acid', 'delamanid', 'bedaquiline', 'linezolid'], 'amplicon': False, 'files': {'ref': 'tbdb.fasta', 'gff': 'tbdb.gff', 'bed': 'tbdb.bed', 'version': 'tbdb.version.json', 'json_db': 'tbdb.dr.json', 'variables': 'tbdb.variables.json', 'spoligotype_spacers': 'tbdb.spoligotype_spacers.txt', 'spoligotype_annotations': 'tbdb.spoligotype_list.csv', 'bedmask': 'tbdb.mask.bed', 'barcode': 'tbdb.barcode.bed'}, 'ref': '/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/share/tbprofiler//tbdb.fasta', 'gff': '/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/share/tbprofiler//tbdb.gff', 'bed': '/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/share/tbprofiler//tbdb.bed', 'version': {'name': 'tbdb', 'commit': 'c2fb9a2', 'Author': 'jodyphelan <jody.phelan@lshtm.ac.uk>', 'Date': 'Tue Oct 4 11:40:15 2022 +0100'}, 'variables': {'snpEff_db': 'Mycobacterium_tuberculosis_h37rv', 'drugs': ['rifampicin', 'isoniazid', 'ethambutol', 'pyrazinamide', 'streptomycin', 'fluoroquinolones', 'moxifloxacin', 'ofloxacin', 'levofloxacin', 'ciprofloxacin', 'aminoglycosides', 'amikacin', 'capreomycin', 'kanamycin', 'cycloserine', 'ethionamide', 'clofazimine', 'para-aminosalicylic_acid', 'delamanid', 'bedaquiline', 'linezolid'], 'amplicon': False, 'files': {'ref': 'tbdb.fasta', 'gff': 'tbdb.gff', 'bed': 'tbdb.bed', 'version': 'tbdb.version.json', 'json_db': 'tbdb.dr.json', 'variables': 'tbdb.variables.json', 'spoligotype_spacers': 'tbdb.spoligotype_spacers.txt', 'spoligotype_annotations': 'tbdb.spoligotype_list.csv', 'bedmask': 'tbdb.mask.bed', 'barcode': 'tbdb.barcode.bed'}}, 'spoligotype_spacers': '/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/share/tbprofiler//tbdb.spoligotype_spacers.txt', 'spoligotype_annotations': '/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/share/tbprofiler//tbdb.spoligotype_list.csv', 'bedmask': '/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/share/tbprofiler//tbdb.mask.bed', 'barcode': '/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/share/tbprofiler//tbdb.barcode.bed'}, 'run_delly': True}

  Traceback:

  File "/etc/ace-data/home/sbabirye/TBProfiler/tb-profiler", line 694, in <module>
  args.func(args)
  File "/etc/ace-data/home/sbabirye/TBProfiler/tb-profiler", line 153, in main_profile
  results.update(pp.run_profiler(args))
  File "/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/lib/python3.10/site-packages/pathogen_profiler-2.0.4-py3.10.egg/pathogenprofiler/cli.py", line 36, in run_profiler
  args.bam_file = get_bam_file(args)
  File "/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/lib/python3.10/site-packages/pathogen_profiler-2.0.4-py3.10.egg/pathogenprofiler/cli.py", line 110, in get_bam_file
  bam_obj = fastq_obj.map_to_ref(
  File "/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/lib/python3.10/site-packages/pathogen_profiler-2.0.4-py3.10.egg/pathogenprofiler/fastq.py", line 110, in map_to_ref
  run_cmd("samtools sort -m %(max_mem)s -n -@ %(threads)s  %(bam_unsort_file)s | samtools fixmate -@ %(threads)s -m - - | samtools sort -m %(max_mem)s -@ %(threads)s - | samtools markdup -@ %(threads)s - %(bam_file)s" % vars(self))
  File "/etc/ace-data/home/sbabirye/.conda/envs/TBProfiler_ENV/lib/python3.10/site-packages/pathogen_profiler-2.0.4-py3.10.egg/pathogenprofiler/utils.py", line 404, in run_cmd
  raise ValueError("Command Failed:\n%s\nstderr:\n%s" % (cmd,stderr.decode()))

  Value:

  
  Command Failed:
  set -u pipefail; samtools sort -m 768M -n -@ 4  ./60ef5aaf-1eb5-4bb7-9ed7-8e628b7f249c.unsort.bam | samtools fixmate -@ 4 -m - - | samtools sort -m 768M -@ 4 - | samtools markdup -@ 4 - ./60ef5aaf-1eb5-4bb7-9ed7-8e628b7f249c.bam
  stderr:
  [bam_sort_core] merging from 0 files and 4 in-memory blocks...

[jodyphelan]

Can you give me some details about the system you are running tb-profiler on? It might be an issue of running out of RAM

[aebordoy]

I think I am facing a similar problem when I try to run one of my samples. However, running the test indicated in the manual did not result in such error:

Command: tb-profiler profile -1 44015MTL22_R1.fastq.gz -2 44015MTL22_R2.fastq.gz -t 4 -p 44015MTL22 --txt

Error: [14:26:19] ERROR [bam_sort_core] merging from 0 files and 4 in-memory blocks... utils.py:485 samtools sort: couldn't allocate memory for bam_mem
samtools markdup: error reading header

Traceback:
Traceback (most recent call last): File "/imppc/labs/emlab/aescalas/miniconda3/envs/TBProfiler/bin/tb-profiler", line 583, in args.func(args) File "/imppc/labs/emlab/aescalas/miniconda3/envs/TBProfiler/bin/tb-profiler", line 101, in main_profile variants_profile = pp.run_profiler(args) File "/imppc/labs/emlab/aescalas/miniconda3/envs/TBProfiler/lib/python3.10/site-packages/pathogenprofiler/cli.py", line 191, in run_profiler args.bam = get_bam_file(args) File "/imppc/labs/emlab/aescalas/miniconda3/envs/TBProfiler/lib/python3.10/site-packages/pathogenprofiler/cli.py", line 255, in get_bam_file bam_obj = fastq_obj.map_to_ref( File "/imppc/labs/emlab/aescalas/miniconda3/envs/TBProfiler/lib/python3.10/site-packages/pathogenprofiler/fastq.py", line 115, in map_to_ref run_cmd("samtools sort -m %(max_mem)s -n -@ %(threads)s %(bam_unsort_file)s | samtools fixmate -@ %(threads)s -m - - | samtools sort -m %(max_mem)s -@ %(threads)s - | samtools markdup -@ %(threads)s - %(bam_file)s" % vars(self)) File "/imppc/labs/emlab/aescalas/miniconda3/envs/TBProfiler/lib/python3.10/site-packages/pathogenprofiler/utils.py", line 486, in run_cmd raise ValueError("Command Failed:\n%s\nstderr:\n%s" % (cmd,result.stderr.decode())) ValueError: Command Failed: /bin/bash -c set -o pipefail; samtools sort -m 768M -n -@ 4 /imppc/labs/emlab/aescalas/TBSEQ/MTB2409/f4aaaead-7bda-4a9b-9ed8-a3dc8132eb70.unsort.bam | samtools fixmate -@ 4 -m - - | samtools sort -m 768M -@ 4 - | samtools markdup -@ 4 - /imppc/labs/emlab/aescalas/TBSEQ/MTB2409/f4aaaead-7bda-4a9b-9ed8-a3dc8132eb70.bam stderr: [bam_sort_core] merging from 0 files and 4 in-memory blocks... samtools sort: couldn't allocate memory for bam_mem samtools markdup: error reading header

Thank you

[jodyphelan]

How much memory do you have on your system? and how big are the fastq files?

[aebordoy]

I'm not sure about the memory available when I run it. I was running it on a server but on my own user space. Executing the same command through qsub solved the problem.

[jodyphelan]

Ok, somtimes with servers using job queues they limit what you can run on your own profile. If it worked when you used qsub then I think there might be a limitation on the resources given to your own user space.

[aebordoy]

Right, I think that was the problem, my bad. As an update, I have been able to run smoothly a batch of about 120 samples! Thanks a lot!