Conda installtion

[mabouelk]

Hi,

Can I use conda for installing this tool? I am not familiar with docker but I have downloaded it and try to follow the instruction but no luck, maybe because I do not know how to adjust the port. I see in the previous issue conda but I don't understand the Italian language.

[yp]

Thanks for your interest about MALVIRUS. Unfortunately, at the moment, MALVIRUS cannot be installed through conda. However, if you want to share with us some details about the error you get, we can provide some assistance in installing MALVIRUS through Docker (and we can also integrate some extra information in the documentation).

[mabouelk]

it stops at this point for more than hour

(base) mcabou:~ moh$ docker run -p 56733:80 --mount type=volume,source=malvirus_jobs,target=/jobs algolab/malvirus Checking for script in /app/prestart.sh Running script /app/prestart.sh Running inside /app/prestart.sh, you could add migrations to this file, e.g.:

! /usr/bin/env bash

Let the DB start

sleep 10;

Run migrations

alembic upgrade head

/usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. 'Supervisord is running as root and it is searching ' 2020-05-21 23:00:07,307 CRIT Supervisor running as root (no user in config file) 2020-05-21 23:00:07,307 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing 2020-05-21 23:00:07,316 INFO RPC interface 'supervisor' initialized 2020-05-21 23:00:07,316 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2020-05-21 23:00:07,316 INFO supervisord started with pid 1 2020-05-21 23:00:08,320 INFO spawned: 'nginx' with pid 12 2020-05-21 23:00:08,321 INFO spawned: 'uwsgi' with pid 13 [uWSGI] getting INI configuration from /app/uwsgi.ini [uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini

;uWSGI instance configuration [uwsgi] cheaper = 2 processes = 16 ini = /app/uwsgi.ini module = main callable = app master = true enable-threads = true touch-reload = /app/uwsgi.ini ini = /etc/uwsgi/uwsgi.ini socket = /tmp/uwsgi.sock chown-socket = nginx:nginx chmod-socket = 664 hook-master-start = unix_signal:15 gracefully_kill_them_all need-app = true die-on-term = true show-config = true ;end of configuration

Starting uWSGI 2.0.18 (64bit) on [Thu May 21 23:00:08 2020] compiled with version: 6.3.0 20170516 on 04 May 2020 14:18:33 os: Linux-4.19.76-linuxkit #1 SMP Fri Apr 3 15:53:26 UTC 2020 nodename: 58bc95e2ea35 machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 6 current working directory: /app detected binary path: /usr/local/bin/uwsgi your memory page size is 4096 bytes detected max file descriptor number: 1048576 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 3 uWSGI running as root, you can use --uid/--gid/--chroot options WARNING: you are running uWSGI as root !!! (use the --uid flag) Python version: 3.7.7 (default, Apr 23 2020, 15:05:00) [GCC 6.3.0 20170516] Python main interpreter initialized at 0x56146163efc0 uWSGI running as root, you can use --uid/--gid/--chroot options WARNING: you are running uWSGI as root !!! (use the --uid flag) python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 1239640 bytes (1210 KB) for 16 cores Operational MODE: preforking WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x56146163efc0 pid: 13 (default app) uWSGI running as root, you can use --uid/--gid/--chroot options WARNING: you are running uWSGI as root !!! (use the --uid flag) uWSGI is running in multiple interpreter mode spawned uWSGI master process (pid: 13) spawned uWSGI worker 1 (pid: 15, cores: 1) spawned uWSGI worker 2 (pid: 16, cores: 1) running "unix_signal:15 gracefully_kill_them_all" (master-start)... 2020-05-21 23:00:09,576 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-05-21 23:00:09,576 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

[yp]

As far as I can tell, this actually means that MALVIRUS is running. Could you open your browser and navigate to address http://localhost:56733 ? (You can also click on this link.)

[mabouelk]

Thank you so much, yes I open the browser and run it. I tried the tutorial files but it failed here is the log:

[Thu May 21 23:22:35 2020] rule kmc: input: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/SRR11514750.fastq.gz output: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers.kmc_pre log: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmc.log, /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmc.json jobid: 3 threads: 4

Job counts: count jobs 1 kmc 1 [Thu May 21 23:22:46 2020] Finished job 3. 1 of 4 steps (25%) done

[Thu May 21 23:22:46 2020] rule malva: input: /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta, /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf, /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers.kmc_pre output: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf log: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.log, /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.json jobid: 2

Job counts: count jobs 1 malva 1 /bin/bash: line 1: 2784 Killed malva-geno -1 -k 35 -r 43 -c 10000 -b 1 /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers > /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf 2> /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.log [Thu May 21 23:23:00 2020] Error in rule malva:  jobid: 0  output: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf  log: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.log, /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.json (check log file(s) for error message)  RuleException: CalledProcessError in line 154 of /snakemake/Snakefile.malva: Command 'malva-geno -1 -k 35 -r 43 -c 10000 -b 1 /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers > /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf 2> /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.log' returned non-zero exit status 137. File "/snakemake/Snakefile.malva", line 164, in __rule_malva File "/snakemake/Snakefile.malva", line 154, in __rule_malva File "/opt/conda/envs/malva-env/lib/python3.7/subprocess.py", line 411, in check_output File "/opt/conda/envs/malva-env/lib/python3.7/subprocess.py", line 512, in run File "/opt/conda/envs/malva-env/lib/python3.7/concurrent/futures/thread.py", line 57, in run Exiting because a job execution failed. Look above for error message Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/.snakemake/log/2020-05-21T232235.798242.snakemake.log

[yp]

Thanks for the report. Could you please attach the content of the log that you obtain at this address?

[mabouelk]

{"alias":"SRR11514750","description":"variant genotyping of individual SRR11514750","filename":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/SRR11514750.fastq.gz","id":"20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79","input":{"gtf":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/sars-cov-2.gff","reference":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta","vcf":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf"},"log":{"last_time":"2020-05-21 23:23:00","status":"Failed","steps":{"KMC":{"command":"kmc -t4 -m4 -k43 -ci5 -cs10000 -fm /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/SRR11514750.fastq.gz /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/tmp &> /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmc.log","config":{"cores":4,"gtf":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/sars-cov-2.gff","lenkmers":43,"malvak":35,"maxmem":4,"maxocc":10000,"minocc":5,"reference":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta","sample":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/SRR11514750.fastq.gz","vcf":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf","workdir":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79"},"input":{"sample":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/SRR11514750.fastq.gz"},"log":"****\n\nStage 1: 0%\nStage 1: 100%\n\nStage 2: 0%\nStage 2: 1%\nStage 2: 2%\nStage 2: 3%\nStage 2: 4%\nStage 2: 5%\nStage 2: 6%\nStage 2: 7%\nStage 2: 8%\nStage 2: 9%\nStage 2: 10%\nStage 2: 11%\nStage 2: 12%\nStage 2: 13%\nStage 2: 14%\nStage 2: 15%\nStage 2: 16%\nStage 2: 17%\nStage 2: 18%\nStage 2: 19%\nStage 2: 20%\nStage 2: 21%\nStage 2: 22%\nStage 2: 23%\nStage 2: 24%\nStage 2: 25%\nStage 2: 26%\nStage 2: 27%\nStage 2: 28%\nStage 2: 29%\nStage 2: 30%\nStage 2: 31%\nStage 2: 32%\nStage 2: 33%\nStage 2: 34%\nStage 2: 35%\nStage 2: 36%\nStage 2: 37%\nStage 2: 38%\nStage 2: 39%\nStage 2: 40%\nStage 2: 41%\nStage 2: 42%\nStage 2: 43%\nStage 2: 44%\nStage 2: 45%\nStage 2: 46%\nStage 2: 47%\nStage 2: 48%\nStage 2: 49%\nStage 2: 50%\nStage 2: 51%\nStage 2: 52%\nStage 2: 53%\nStage 2: 54%\nStage 2: 55%\nStage 2: 56%\nStage 2: 57%\nStage 2: 58%\nStage 2: 59%\nStage 2: 60%\nStage 2: 61%\nStage 2: 62%\nStage 2: 63%\nStage 2: 64%\nStage 2: 65%\nStage 2: 66%\nStage 2: 67%\nStage 2: 68%\nStage 2: 69%\nStage 2: 70%\nStage 2: 71%\nStage 2: 72%\nStage 2: 73%\nStage 2: 74%\nStage 2: 75%\nStage 2: 76%\nStage 2: 77%\nStage 2: 78%\nStage 2: 79%\nStage 2: 80%\nStage 2: 81%\nStage 2: 82%\nStage 2: 83%\nStage 2: 84%\nStage 2: 85%\nStage 2: 86%\nStage 2: 87%\nStage 2: 88%\nStage 2: 89%\nStage 2: 90%\nStage 2: 91%\nStage 2: 92%\nStage 2: 93%\nStage 2: 94%\nStage 2: 95%\nStage 2: 96%\nStage 2: 97%\nStage 2: 98%\nStage 2: 99%\nStage 2: 100%\n1st stage: 5.7723s\n2nd stage: 4.68519s\nTotal : 10.4575s\nTmp size : 112MB\n\nStats:\n No. of k-mers below min. threshold : 15740125\n No. of k-mers above max. threshold : 0\n No. of unique k-mers : 17100708\n No. of unique counted k-mers : 1360583\n Total no. of k-mers : 121139100\n Total no. of sequences : 0\n Total no. of super-k-mers : 6909420\n","output":{"kmc_out":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers.kmc_pre"},"params":{"kmc_prefix":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers","kmc_tmp":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/tmp","lenkmers":43,"maxmem":4,"maxocc":10000,"minocc":5},"result":"Success","return_code":0,"time":"2020-05-21 23:22:46"},"malva":{"command":"malva-geno -1 -k 35 -r 43 -c 10000 -b 1 /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers > /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf 2> /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.log","config":{"cores":4,"gtf":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/sars-cov-2.gff","lenkmers":43,"malvak":35,"maxmem":4,"maxocc":10000,"minocc":5,"reference":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta","sample":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/SRR11514750.fastq.gz","vcf":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf","workdir":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79"},"input":{"kmc":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers.kmc_pre","ref":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta","vcf":"/jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf"},"log":"[malva-geno/Reference parsing] Time elapsed 0s\n[malva-geno/Reference processed] Time elapsed 0s\n[malva-geno/VCF parsing (Bloom Filter construction)] Time elapsed 0s\n[malva-geno/Processed 1156 variants] Time elapsed 2s\n[malva-geno/BF creation complete] Time elapsed 7s\n[malva-geno/Reference BF construction] Time elapsed 7s\n[malva-geno/Reference BF creation complete] Time elapsed 7s\n","output":{"vcf":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf"},"params":{"bfsize":1,"c":10000,"k":35,"kmc_prefix":"/jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers","r":43},"result":"Failed","return_code":137,"time":"2020-05-21 23:23:00"}}},"params":{"cores":"4","lenkmers":"43","malvak":"35","maxmem":"4","maxocc":"10000","minocc":"5","vcf":"NCBI-SARS-CoV-2-20200430-181053"},"snakemake":"Building DAG of jobs...\nUsing shell: /bin/bash\nProvided cores: 4\nRules claiming more threads will be scaled down.\nJob counts:\n\tcount\tjobs\n\t1\tkmc\n\t1\tmalva\n\t1\trun\n\t1\ttag_malva_vcf\n\t4\n\n[Thu May 21 23:22:35 2020]\nrule kmc:\n input: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/SRR11514750.fastq.gz\n output: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers.kmc_pre\n log: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmc.log, /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmc.json\n jobid: 3\n threads: 4\n\n\u001b[33mJob counts:\n\tcount\tjobs\n\t1\tkmc\n\t1\u001b[0m\n[Thu May 21 23:22:46 2020]\nFinished job 3.\n1 of 4 steps (25%) done\n\n[Thu May 21 23:22:46 2020]\nrule malva:\n input: /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta, /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf, /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers.kmc_pre\n output: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf\n log: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.log, /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.json\n jobid: 2\n\n\u001b[33mJob counts:\n\tcount\tjobs\n\t1\tmalva\n\t1\u001b[0m\n/bin/bash: line 1: 2784 Killed malva-geno -1 -k 35 -r 43 -c 10000 -b 1 /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers > /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf 2> /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.log\n\u001b[32m[Thu May 21 23:23:00 2020]\u001b[0m\n\u001b[31mError in rule malva:\u001b[0m\n\u001b[31m jobid: 0\u001b[0m\n\u001b[31m output: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf\u001b[0m\n\u001b[31m log: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.log, /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.json (check log file(s) for error message)\u001b[0m\n\u001b[31m\u001b[0m\n\u001b[31mRuleException:\nCalledProcessError in line 154 of /snakemake/Snakefile.malva:\nCommand 'malva-geno -1 -k 35 -r 43 -c 10000 -b 1 /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/reference.fasta /jobs/vcf/NCBI-SARS-CoV-2-20200430-181053/run.cleaned.vcf /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/KMC/kmers > /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.non_tagged.vcf 2> /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/malva/malva.log' returned non-zero exit status 137.\n File \"/snakemake/Snakefile.malva\", line 164, in __rule_malva\n File \"/snakemake/Snakefile.malva\", line 154, in __rule_malva\n File \"/opt/conda/envs/malva-env/lib/python3.7/subprocess.py\", line 411, in check_output\n File \"/opt/conda/envs/malva-env/lib/python3.7/subprocess.py\", line 512, in run\n File \"/opt/conda/envs/malva-env/lib/python3.7/concurrent/futures/thread.py\", line 57, in run\u001b[0m\n\u001b[31mExiting because a job execution failed. Look above for error message\u001b[0m\nShutting down, this might take some time.\nExiting because a job execution failed. Look above for error message\nComplete log: /jobs/malva/20200521-232235_b7c6d257-fdeb-40ff-904f-2e774e727a79/.snakemake/log/2020-05-21T232235.798242.snakemake.log\n","submission_time":1590103355}

[ldenti]

Hi, thanks for the log. We ran MALVIRUS on the same data but it completed its execution with success, so we could not replicate your error.

From the logs you sent, we see that malva-geno got killed when he tries to allocate a vector and this could be a memory issue.

How much RAM do you have? Did you limit docker maximum RAM somehow?

Best, Luca

[yp]

Hi @mabouelk, any update on this issue?

[mabouelk]

Hi @yp thank you for your followup. I will try to run it on a workstation with bigger memory.