search

jiarong / VirSorter2

customizable pipeline to identify viral sequences from (meta)genomic data
GNU General Public License v2.0
217 stars 29 forks source link

The output files don't update #198

Open Betula0422 opened 4 months ago

Betula0422 commented 4 months ago

Hi there, I have been successfully run the script with your test dataset within two days. But when I use my data, my output files have not been updated for a week, but there are no errors in the log file. I can't figure out what happened. There are 46,200,228 contigs in my data. Is it because the data is too large? Looking forward to your reply.

Here is my script:

!/bin/bash

source activate base conda activate vs virsorter run -w /data2/Betula/virus/virsorter2/test.out2 -i /data1/wxh/contigs.rnm.fa -d ~/database/virsorter2/db --min-length 5000 -j 28 --include-groups dsDNAphage,ssDNA --min-score 0.5 --use-conda-off all

Here is my log:

[2024-04-30 15:11 INFO] VirSorter 2.2.4 [2024-04-30 15:11 INFO] /data2/Betula/miniconda3/envs/vs/bin/virsorter run -w /data2/Betula/virus/virsorter2/test.out -i /data1/wxh/contigs.rnm.fa -d /data2/Betula/database/virsorter2/db --min-length 1500 -j 28 --use-conda-off all [2024-04-30 15:11 INFO] Using /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/template-config.yaml as config template [2024-04-30 15:11 INFO] conig file written to /data2/Betula/virus/virsorter2/test.out/config.yaml

[2024-04-30 15:11 INFO] Executing: snakemake --snakefile /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/Snakefile --directory /data2/Betula/virus/virsorter2/test.out --jobs 28 --configfile /data2/Betula/virus/virsorter2/test.out/config.yaml --latency-wait 600 --rerun-incomplete --nolock --quiet all
[2024-04-30 15:18 INFO] # of seqs < 1500 bp and removed: 43178270 [2024-04-30 15:18 INFO] # of circular seqs: 1017 [2024-04-30 15:18 INFO] # of linear seqs : 3020941 [2024-04-30 15:19 INFO] Finish spliting circular contig file with common rbs [2024-04-30 15:20 INFO] Finish spliting linear contig file with common rbs [2024-04-30 16:26 INFO] Step 1 - preprocess finished. [2024-05-05 17:10 INFO] Step 2 - extract-feature finished.

jiarong commented 4 months ago

Hi, your data is quite large. I see the step 2 (the longest step) was finished a week ago. The last step should NOT take this long. Can you check which process it is stuck at? (using top command). Also note that if you this run errors out, you can just rerun the same command, VirSorter2 can pick up where it stops and continue.

Betula0422 commented 4 months ago

Thank you for your reply! I don't know why I can't reply at github.

I used "top" command. The results are as follows.

top - 21:14:47 up 43 days, 7:49, 2 users, load average: 28.00, 28.04, 28.05

Tasks: 433 total, 31 running, 402 sleeping, 0 stopped, 0 zombie

%Cpu(s): 33.1 us, 2.8 sy, 0.0 ni, 64.0 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st

KiB Mem : 65439004 total, 41670432 free, 20060116 used, 3708456 buff/cache

KiB Swap: 67108860 total, 66984144 free, 124716 used. 44642244 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

226414 Betula 20 0 966024 751388 21308 R 50.2 1.1 4710:43 python

226392 Betula 20 0 966284 731720 21308 R 49.8 1.1 4726:53 python

226426 Betula 20 0 966028 731636 21308 R 49.2 1.1 4722:23 python

226386 Betula 20 0 900868 686108 21308 R 48.8 1.0 4718:59 python

226421 Betula 20 0 933512 718840 21308 R 34.7 1.1 4718:42 python

226400 Betula 20 0 966028 731604 21308 R 34.3 1.1 4714:19 python

226390 Betula 20 0 966028 731396 21308 R 33.7 1.1 4722:25 python

226394 Betula 20 0 966028 732992 21308 R 33.3 1.1 4715:13 python

226396 Betula 20 0 966024 751392 21308 R 33.3 1.1 4715:17 python

226398 Betula 20 0 900480 685944 21308 R 33.3 1.0 4720:18 python

226402 Betula 20 0 966412 751652 21308 R 33.3 1.1 4720:49 python

226404 Betula 20 0 966028 731552 21308 R 33.3 1.1 4714:23 python

226408 Betula 20 0 966028 731584 21308 R 33.3 1.1 4728:26 python

226412 Betula 20 0 900484 683500 21308 R 33.3 1.0 4720:43 python

226416 Betula 20 0 900484 661692 21308 R 33.3 1.0 4718:54 python

226424 Betula 20 0 900616 664976 21308 R 33.3 1.0 4721:11 python

226429 Betula 20 0 966408 751828 21308 R 33.3 1.1 4713:25 python

226430 Betula 20 0 966416 731820 21308 R 33.3 1.1 4719:45 python

226432 Betula 20 0 900484 665912 21308 R 33.3 1.0 4725:41 python

226436 Betula 20 0 900484 638932 21308 R 33.3 1.0 4714:19 python

59943 Betula 20 0 795176 583612 24792 R 33.0 0.9 4568:41 python

226388 Betula 20 0 966024 751460 21308 R 33.0 1.1 4721:37 python

226407 Betula 20 0 900740 666020 21308 R 33.0 1.0 4719:48 python

226410 Betula 20 0 900740 686160 21308 R 33.0 1.0 4716:02 python

226418 Betula 20 0 900480 685960 21308 R 33.0 1.0 4720:03 python

226422 Betula 20 0 966024 751404 21308 R 33.0 1.1 4716:34 python

226434 Betula 20 0 900484 658764 21308 R 33.0 1.0 4714:36 python

226511 Betula 20 0 729884 514924 21304 R 33.0 0.8 4658:24 python

70311 root 20 0 44924 1388 1088 R 3.0 0.0 0:00.09 ps

69994 Betula 20 0 158008 2848 1816 R 0.7 0.0 0:00.62 top

I also used "ps ux" command. The results are as follows.

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

Betula 59942 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/ssDNA

Betula 59943 35.2 0.8 761888 531292 ? R May05 4566:51 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/ssDNA/all.p

Betula 68996 0.0 0.0 156376 2452 ? S 21:09 0:00 sshd: @.***/1

Betula 68997 0.7 0.0 116968 3564 pts/1 Ss 21:09 0:00 -bash

Betula 69129 0.0 0.0 151068 2116 pts/1 R+ 21:09 0:00 ps ux

Betula 103797 0.0 0.0 115384 1736 ? Ss Apr30 0:00 -bash

Betula 103994 0.0 0.0 113136 1552 ? S Apr30 0:00 /bin/bash /opt/gridview//pbs/dispatcher/mom_priv/jobs/3804.node200.SC

Betula 104000 0.0 0.0 154096 24108 ? S Apr30 0:00 /data2/Betula/miniconda3/envs/vs/bin/python3.8 /data2/Betula/miniconda3/envs/vs/bin/virsorter run -w /

Betula 104018 0.0 0.1 3482232 112800 ? Sl Apr30 1:05 /data2/Betula/miniconda3/envs/vs/bin/python3.8 /data2/Betula/miniconda3/envs/vs/bin/snakemake --snakef

Betula 226385 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226386 35.7 1.0 900740 658348 ? R May05 4717:15 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226387 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226388 35.7 1.0 900480 685980 ? R May05 4719:52 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226389 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226390 35.7 1.1 966024 751156 ? R May05 4720:41 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226391 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226392 35.8 1.0 900484 686032 ? R May05 4725:04 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226393 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226394 35.7 1.1 966024 751580 ? R May05 4713:30 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226395 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226396 35.7 1.0 900480 685916 ? R May05 4713:28 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226397 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226398 35.7 1.0 900480 685996 ? R May05 4718:37 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226399 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226400 35.7 1.0 933256 702432 ? R May05 4712:32 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226401 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226402 35.7 1.1 966412 751652 ? R May05 4719:03 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226403 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226404 35.7 1.1 966024 751368 ? R May05 4712:38 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226405 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226406 0.0 0.0 113136 1452 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226407 35.7 1.1 966280 751524 ? R May05 4718:06 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226408 35.8 1.1 966024 751496 ? R May05 4726:45 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226409 0.0 0.0 113136 1452 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226410 35.7 1.1 966284 731804 ? R May05 4714:20 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226411 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226412 35.7 1.1 966028 731604 ? R May05 4718:58 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226413 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226414 35.6 1.0 900484 665044 ? R May05 4708:58 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226415 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226416 35.7 1.0 900480 685976 ? R May05 4717:13 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226417 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226418 35.7 1.1 966024 751432 ? R May05 4718:23 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226419 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226420 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226421 35.7 1.1 966280 751660 ? R May05 4716:59 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226422 35.7 1.0 900484 666132 ? R May05 4714:47 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226423 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226424 35.7 1.0 933388 715440 ? R May05 4719:27 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226425 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226426 35.7 1.1 966024 751548 ? R May05 4720:36 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226427 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226428 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226429 35.7 1.1 966408 751672 ? R May05 4711:42 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226430 35.7 1.1 966156 751564 ? R May05 4718:00 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226431 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226432 35.7 1.0 900484 663240 ? R May05 4723:58 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226433 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226434 35.7 1.1 966028 731584 ? R May05 4712:46 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226435 0.0 0.0 113136 1452 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA

Betula 226436 35.7 1.0 900480 685924 ? R May05 4712:33 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/

Betula 226510 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/ssDNA

Betula 226511 35.2 0.8 794912 579948 ? R May05 4656:40 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/ssDNA/all.p

At 2024-05-14 20:15:10, "jiarong" @.***> wrote:

Hi, your data is quite large. I see the step 2 (the longest step) was finished a week ago. The last step should NOT take this long. Can you check which process it is stuck at? (using top command). Also note that if you this run errors out, you can just rerun the same command, VirSorter2 can pick up where it stops and continue.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

jiarong commented 4 months ago

It shows that VirSorter2 is still processing the prophage extraction step, NOT stuck due to errors. For future runs, you can split the large input file and process them in parallel.

Betula0422 commented 4 months ago

Thank you so much for your reply. I got it! Good luck with your job!

---- Replied Message ---- | From | @.> | | Date | 05/15/2024 10:25 | | To | jiarong/VirSorter2 @.> | | Cc | Betula0422 @.>, Author @.> | | Subject | Re: [jiarong/VirSorter2] The output files don't update (Issue #198) |

It shows that VirSorter2 is still processing the prophage extraction step, NOT stuck due to errors. For future runs, you can split the large input file and process them in parallel.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>