ntm-profiler running failed in docker container

[yujun2017]

We can't run NTM-profiler 0.3.0 due to update_db failed.

when we tried to run NTM-profiler 0.2.0 inside docker container, we got the error. I have attached error text file in the tickets. Could you please let us know how to fix the issues?

Running command: set -u pipefail; kmc -t1 -sf1 -sp1 -sr1 -k31 @171cd1ce-30eb-4cff-b32f-2a8032c51490.list 171cd1ce-30eb-4cff-b32f-2a8032c51490 171cd1ce-30eb-4cff-b32f-2a8032c51490 Traceback (most recent call last): File "/opt/conda/envs/ntm-profiler/bin/ntm-profiler", line 322, in args.func(args) File "/opt/conda/envs/ntm-profiler/bin/ntm-profiler", line 89, in main_profile species_prediction = pp.speciate(args) File "/opt/conda/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/cli.py", line 64, in speciate kmer_dump = fastq_class.get_kmer_counts(args.files_prefix,threads=args.threads) File "/opt/conda/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/fastq.py", line 127, in get_kmer_counts run_cmd(f"kmc {bins} -t{threads} -sf{threads} -sp{threads} -sr{threads} -k{klen} @{tmp_file_list} {tmp_prefix} {tmp_prefix}") File "/opt/conda/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/utils.py", line 391, in run_cmd raise ValueError("Command Failed:\n%s\nstderr:\n%s" % (cmd,stderr.decode())) ValueError: Command Failed: set -u pipefail; kmc -t1 -sf1 -sp1 -sr1 -k31 @171cd1ce-30eb-4cff-b32f-2a8032c51490.list 171cd1ce-30eb-4cff-b32f-2a8032c51490 171cd1ce-30eb-4cff-b32f-2a8032c51490

61ae2b93-2dc9-4966-8101-f64c6158c40f.errlog.txt

[jodyphelan]

Hi @yujun2017 ,

Do you have the error report for v0.3.0?

[yujun2017]

Jody: Yes. Here is the error file when I tried to run update_db in 0.3.0 in my mac pro. We would like to use the 0.3.0 version in our application. but we run into errors and try to solve it before we can push to prod. I have another question regarding the ntm-db location. When we push an application to our prod, where should we put the ntm-db? I just noticed ntm-db will be created in the location that we are running the update_db command. When we are using AWS, how do we solve this problem?

Thanks!

John

On Thu, 14 Dec 2023 at 18:10, Jody Phelan @.***> wrote:

Hi @yujun2017 https://github.com/yujun2017 ,

Do you have the error report for v0.3.0?

— Reply to this email directly, view it on GitHub https://github.com/jodyphelan/NTM-Profiler/issues/33#issuecomment-1856908002, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGDIGTFJ7BDDCPAB6G4YDV3YJOBM3AVCNFSM6AAAAABAVJXANCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWHEYDQMBQGI . You are receiving this because you were mentioned.Message ID: @.***>

--

John Yu

ntm-profiler error report

• OS: darwin
• ntm-profiler version: 0.3.0
• pathogen-profiler version: 3.1.0
• Program call:

  {'no_clean': False, 'logging': 'INFO', 'repo': 'https://github.com/ntm-db/ntm-db.git', 'branch': 'master', 'commit': None, 'dir': '.', 'temp': '.', 'func': <function main_update_db at 0x109b35ea0>, 'software_name': 'ntm-profiler', 'tmp_prefix': 'b8f464ca-37e6-4940-a5cd-62a3651e5459', 'files_prefix': './b8f464ca-37e6-4940-a5cd-62a3651e5459'}

  Traceback:

  File "/usr/local/conda/anaconda3/envs/ntm-profiler/bin/ntm-profiler", line 390, in <module>
  args.func(args)
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/bin/ntm-profiler", line 216, in main_update_db
  pp.run_cmd(f'ntm-profiler create_resistance_db --prefix {species.replace(" ","_")} --csv resistance_variants.csv {barcode_arg} --load')
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/utils.py", line 392, in run_cmd
  raise ValueError("Command Failed:\n%s\nstderr:\n%s" % (cmd,result.stderr.decode()))

  Value:

  
  Command Failed:
  /bin/bash -c set -o pipefail; ntm-profiler create_resistance_db --prefix Mycobacterium_intracellulare_subsp._chimaera --csv resistance_variants.csv --barcode barcode.bed --load
  stderr:
  Traceback (most recent call last):
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/bin/ntm-profiler", line 390, in <module>
  args.func(args)
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/bin/ntm-profiler", line 177, in create_resistance_db
  pp.create_db(args,extra_files=extra_files)
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/db.py", line 418, in create_db
  mutation_lookup = get_snpeff_formated_mutation_list(hgvs_variants,"genome.fasta","genome.gff",json.load(open("variables.json"))["snpEff_db"])
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/db.py", line 193, in get_snpeff_formated_mutation_list
  converted_mutations = verify_mutation_list(hgvs_variants,genes,refseq, snpEffDB)
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/hgvs.py", line 322, in verify_mutation_list
  mutation_conversion = get_ann(mutations_genome,snpEffDB)
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/hgvs.py", line 200, in get_ann
  for l in cmd_out(f"snpEff ann {snpEffDB} {uuid}"):
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/utils.py", line 398, in cmd_out
  run_cmd(cmd)
  File "/usr/local/conda/anaconda3/envs/ntm-profiler/lib/python3.10/site-packages/pathogenprofiler/utils.py", line 392, in run_cmd
  raise ValueError("Command Failed:\n%s\nstderr:\n%s" % (cmd,result.stderr.decode()))
  ValueError: Command Failed:
  /bin/bash -c set -o pipefail; snpEff ann Mycobacterium_chimaera_ZUERICH-1 5753bf95-2f43-475c-8c25-e3c941e7a2a6 > bc4ae575-5929-46b6-adc2-52087cf8bd56
  stderr:
  java.lang.RuntimeException: Property: 'Mycobacterium_chimaera_ZUERICH-1.genome' not found
  at org.snpeff.interval.Genome.<init>(Genome.java:104)
  at org.snpeff.snpEffect.Config.readGenomeConfig(Config.java:784)
  at org.snpeff.snpEffect.Config.readConfig(Config.java:751)
  at org.snpeff.snpEffect.Config.init(Config.java:529)
  at org.snpeff.snpEffect.Config.<init>(Config.java:116)
  at org.snpeff.SnpEff.loadConfig(SnpEff.java:429)
  at org.snpeff.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:889)
  at org.snpeff.snpEffect.commandLine.SnpEffCmdEff.run(SnpEffCmdEff.java:875)
  at org.snpeff.SnpEff.run(SnpEff.java:1141)
  at org.snpeff.SnpEff.main(SnpEff.java:160)

Cleaning up after failed run

[yujun2017]

Jody: I also tried to install ntm-profiler 0.3.0 in the miniconda3 docker image, ntm-profiler can't be updated db. here is the part of dockerfile, You can try

Miniconda Image: Install Conda package

FROM continuumio/miniconda3:latest as conda

RUN conda config --add channels defaults && \ conda config --add channels bioconda && \ conda config --add channels conda-forge && \ conda create -n mykrobe -y -c bioconda mykrobe && \ conda install -c conda-forge zip && \ conda create -n tb-profiler tb-profiler=4.4.2 cryptography=39.0.1 && \

conda create -n ntm-profiler ntm-profiler=0.2.0 cryptography=39.0.1

conda create -n ntm-profiler ntm-profiler=0.3.0

ENV PATH="$PATH:/opt/conda/envs/ntm-profiler/bin" RUN /opt/conda/envs/ntm-profiler/bin/ntm-profiler update_db

John

On Thu, 14 Dec 2023 at 18:10, Jody Phelan @.***> wrote:

Hi @yujun2017 https://github.com/yujun2017 ,

Do you have the error report for v0.3.0?

— Reply to this email directly, view it on GitHub https://github.com/jodyphelan/NTM-Profiler/issues/33#issuecomment-1856908002, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGDIGTFJ7BDDCPAB6G4YDV3YJOBM3AVCNFSM6AAAAABAVJXANCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWHEYDQMBQGI . You are receiving this because you were mentioned.Message ID: @.***>

--

John Yu

[yujun2017]

Jody: This is John Yu from BCBB/NIH and I just want to check with you what's the status for the NTM profiler? We plan to move NTM profiler to Prod in February and hope you can fix the issue before we move on.

Thanks!

John

On Fri, 15 Dec 2023 at 06:56, John Yu @.***> wrote:

Jody: I also tried to install ntm-profiler 0.3.0 in the miniconda3 docker image, ntm-profiler can't be updated db. here is the part of dockerfile, You can try

Miniconda Image: Install Conda package

FROM continuumio/miniconda3:latest as conda

RUN conda config --add channels defaults && \ conda config --add channels bioconda && \ conda config --add channels conda-forge && \ conda create -n mykrobe -y -c bioconda mykrobe && \ conda install -c conda-forge zip && \ conda create -n tb-profiler tb-profiler=4.4.2 cryptography=39.0.1 && \

conda create -n ntm-profiler ntm-profiler=0.2.0 cryptography=39.0.1

conda create -n ntm-profiler ntm-profiler=0.3.0

ENV PATH="$PATH:/opt/conda/envs/ntm-profiler/bin" RUN /opt/conda/envs/ntm-profiler/bin/ntm-profiler update_db

John

On Thu, 14 Dec 2023 at 18:10, Jody Phelan @.***> wrote:

Hi @yujun2017 https://github.com/yujun2017 ,

Do you have the error report for v0.3.0?

— Reply to this email directly, view it on GitHub https://github.com/jodyphelan/NTM-Profiler/issues/33#issuecomment-1856908002, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGDIGTFJ7BDDCPAB6G4YDV3YJOBM3AVCNFSM6AAAAABAVJXANCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWHEYDQMBQGI . You are receiving this because you were mentioned.Message ID: @.***>

--

John Yu

--

John Yu

[yujun2017]

Jody: I just want to update the issue we have found for latest ntm-profiler. The issue is only "ntm-profiler update_db" command. As soon as we created db, we can run ntm-profiler without any issue.