search

epruesse / SINA

SINA - Reference based multiple sequence alignment
https://sina.readthedocs.io
GNU General Public License v3.0
41 stars 4 forks source link

Problem regarding sina #53

Closed debjitde01 closed 5 years ago

debjitde01 commented 5 years ago

I am working with sina and this is the script which i have used it takes too much time still it is not complete. is it running ok?

(sina) pkd@pkd-HP-406-G1-MT:~$ sina -i ~/Sina/amplicons_seeds.fasta --intype fasta -o sina_out --outtype fasta --search --meta-fmt csv --overhang remove --insertion forbid --filter none --fs-kmer-no-fast --fs-kmer-len 10 --fs-req 2 --fs-req-full 1 --fs-min 40 --fs-max 40 --fs-weight 1 --fs-full-len 350 --fs-msc 0.7 --match-score 1 --mismatch-score -1 --pen-gap 5 --pen-gapext 2 --search-cover query --search-iupac optimistic --search-min-sim 0.9 --turn all --lca-quorum 0.7 --search-db ~/Sina/sina_ssu.arb --db ~/Sina/sina_ssu.arb --lca-fields tax_slv [2019-02-09 13:54:08.891] [log] [info] Loglevel set to warning 13:54:08 [SINA] This is SINA 1.4.0.

0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----|

ARB: Loading '/home/pkd/Sina/sina_ssu.arb.index.arb' ARB: no FastLoad File '/home/pkd/Sina/sina_ssu.arb.index.ARM' found => loading entire DB ARB: Loading '/home/pkd/Sina/sina_ssu.arb.index.arb' done

Progress: Remove unused database entries ...................................................................... [100.0%] used: 5s [done] Database contains 38792 species Progress: Preparing sequence data ...................................................................... [100.0%] used: 4s [done]

All species contain data in alignment 'ali_16s'. [ptserver '-build_clean' took 10s] sum of above: 0 b overall alloc: 0 b sum of above: 0 b overall alloc: 0 b ARB: Loading '/home/pkd/Sina/sina_ssu.arb.index.arb' ARB: Opening FastLoad File '/home/pkd/Sina/sina_ssu.arb.index.ARM' ... ARB: Loading '/home/pkd/Sina/sina_ssu.arb.index.arb' done

Building PT-Server for alignment 'ali_16s'... Database contains 38792 species Progress: Checking data ...................................................................... [100.0%] used: 0s [done] Failed to write to '/home/pkd/anaconda3/envs/sina/lib/arb/lib/pts/ptserver.log' Visible memory: 11.7 Gb Restricting used memory (by internal default '90%') to 10.5 Gb Note: Setting envar ARB_MEMORY will override that restriction (percentage or absolute memsize) Memory available for build: 10.4 Gb Estimated memory usage for 1 pass: 752 Mb Overall bases: 54.1 Mbp Max. partition size: 54.1 Mbp (=100.0%) Progress: Build index in 1 passes {pass 1/1}............................................................ [ 25.0%] left: 33s ...................................................................... [ 50.0%] left: 25s ...................................................................... [ 75.0%] left: 13s ...................................................................... [100.0%] used: 1m0s [done] PT_SERVER database "/home/pkd/Sina/sina_ssu.arb.index.arb" has been created. Failed to write to '/home/pkd/anaconda3/envs/sina/lib/arb/lib/pts/ptserver.log' [ptserver '-build' took 1m2s] arb_message: PT_SERVER database "/home/pkd/Sina/sina_ssu.arb.index.arb" has been created. blocksize: 9 allocated: 304 kb [~ 0%] <~33.8 kBlocks [PT1_EMPTY_NODE_SIZE] blocksize: 10 allocated: 515 kb [~ 0%] <~51.5 kBlocks blocksize: 11 allocated: 759 kb [~ 0%] <~69.0 kBlocks [PT1_MIN_CHAIN_ENTRY_SIZE] blocksize: 12 allocated: 735 kb [~ 0%] <~61.3 kBlocks blocksize: 13 allocated: 650 kb [~ 0%] <~50.0 kBlocks blocksize: 14 allocated: 630 kb [~ 0%] <~45.0 kBlocks blocksize: 15 allocated: 56.2 Mb [~11%] <~3.75 MBlocks [PT1_EMPTY_LEAF_SIZE] blocksize: 16 allocated: 532 kb [~ 0%] <~33.3 kBlocks blocksize: 17 allocated: 208 Mb [~41%] <~12.3 MBlocks [PT1_NODE_WITHSONS_SIZE(1)] blocksize: 18 allocated: 477 kb [~ 0%] <~26.6 kBlocks blocksize: 19 allocated: 451 kb [~ 0%] <~23.8 kBlocks [PT1_CHAIN_SHORT_HEAD_SIZE] blocksize: 20 allocated: 415 kb [~ 0%] <~20.8 kBlocks blocksize: 21 allocated: 389 kb [~ 0%] <~18.6 kBlocks [PT1_CHAIN_LONG_HEAD_SIZE] blocksize: 22 allocated: 330 kb [~ 0%] <~15.1 kBlocks blocksize: 23 allocated: 259 kb [~ 0%] <~11.3 kBlocks [PT1_MAX_CHAIN_ENTRY_SIZE] blocksize: 24 allocated: 78.0 kb [~ 0%] <~3.25 kBlocks blocksize: 25 allocated: 69.7 Mb [~14%] <~2.79 MBlocks [PT1_NODE_WITHSONS_SIZE(2)] blocksize: 26 allocated: 221 kb [~ 0%] <~8.51 kBlocks blocksize: 27 allocated: 385 kb [~ 0%] <~14.3 kBlocks blocksize: 28 allocated: 539 kb [~ 0%] <~19.3 kBlocks blocksize: 29 allocated: 14.6 kb [~ 0%] <~ 512 Blocks blocksize: 30 allocated: 750 kb [~ 0%] <~25.1 kBlocks blocksize: 31 allocated: 465 kb [~ 0%] <~15.1 kBlocks blocksize: 32 allocated: 200 kb [~ 0%] <~6.25 kBlocks blocksize: 33 allocated: 26.0 Mb [~ 5%] <~ 807 kBlocks [PT1_NODE_WITHSONS_SIZE(3)] blocksize: 34 allocated: 8.51 kb [~ 0%] <~ 256 Blocks blocksize: 35 allocated: 403 kb [~ 0%] <~11.6 kBlocks blocksize: 36 allocated: 630 kb [~ 0%] <~17.6 kBlocks blocksize: 37 allocated: 204 kb [~ 0%] <~5.50 kBlocks blocksize: 38 allocated: 266 kb [~ 0%] <~7.00 kBlocks blocksize: 39 allocated: 9.76 kb [~ 0%] <~ 256 Blocks blocksize: 40 allocated: 520 kb [~ 0%] <~13.1 kBlocks blocksize: 41 allocated: 113 Mb [~22%] <~2.76 MBlocks [PT1_NODE_WITHSONS_SIZE(4)] blocksize: 42 allocated: 273 kb [~ 0%] <~6.50 kBlocks blocksize: 43 allocated: 151 kb [~ 0%] <~3.50 kBlocks blocksize: 44 allocated: 11.1 kb [~ 0%] <~ 256 Blocks blocksize: 45 allocated: 304 kb [~ 0%] <~6.75 kBlocks blocksize: 46 allocated: 552 kb [~ 0%] <~12.1 kBlocks blocksize: 47 allocated: 317 kb [~ 0%] <~6.75 kBlocks blocksize: 48 allocated: 252 kb [~ 0%] <~5.25 kBlocks blocksize: 49 allocated: 3.62 Mb [~ 1%] <~75.5 kBlocks [PT1_NODE_WITHSONS_SIZE(5)] blocksize: 50 allocated: 113 kb [~ 0%] <~2.25 kBlocks blocksize: 51 allocated: 179 kb [~ 0%] <~3.50 kBlocks blocksize: 52 allocated: 312 kb [~ 0%] <~6.00 kBlocks blocksize: 53 allocated: 199 kb [~ 0%] <~3.75 kBlocks blocksize: 54 allocated: 13.6 kb [~ 0%] <~ 256 Blocks blocksize: 55 allocated: 206 kb [~ 0%] <~3.75 kBlocks blocksize: 56 allocated: 98.0 kb [~ 0%] <~1.75 kBlocks blocksize: 57 allocated: 941 kb [~ 0%] <~16.6 kBlocks [PT1_NODE_WITHSONS_SIZE(6)] blocksize: 58 allocated: 479 kb [~ 0%] <~8.26 kBlocks blocksize: 60 allocated: 450 kb [~ 0%] <~7.50 kBlocks blocksize: 61 allocated: 290 kb [~ 0%] <~4.75 kBlocks blocksize: 62 allocated: 93.0 kb [~ 0%] <~1.50 kBlocks blocksize: 63 allocated: 63.0 kb [~ 0%] <~1.00 kBlocks blocksize: 65 allocated: 146 kb [~ 0%] <~2.25 kBlocks blocksize: 66 allocated: 215 kb [~ 0%] <~3.25 kBlocks blocksize: 67 allocated: 251 kb [~ 0%] <~3.75 kBlocks blocksize: 68 allocated: 136 kb [~ 0%] <~2.00 kBlocks blocksize: 70 allocated: 140 kb [~ 0%] <~2.00 kBlocks blocksize: 71 allocated: 142 kb [~ 0%] <~2.00 kBlocks blocksize: 72 allocated: 180 kb [~ 0%] <~2.50 kBlocks blocksize: 73 allocated: 383 kb [~ 0%] <~5.25 kBlocks blocksize: 75 allocated: 206 kb [~ 0%] <~2.75 kBlocks blocksize: 76 allocated: 209 kb [~ 0%] <~2.75 kBlocks blocksize: 77 allocated: 366 kb [~ 0%] <~4.75 kBlocks blocksize: 78 allocated: 176 kb [~ 0%] <~2.25 kBlocks blocksize: 80 allocated: 80.0 kb [~ 0%] <~1.00 kBlocks blocksize: 81 allocated: 40.5 kb [~ 0%] <~ 512 Blocks blocksize: 82 allocated: 82.0 kb [~ 0%] <~1.00 kBlocks blocksize: 83 allocated: 166 kb [~ 0%] <~2.00 kBlocks blocksize: 85 allocated: 234 kb [~ 0%] <~2.75 kBlocks blocksize: 86 allocated: 172 kb [~ 0%] <~2.00 kBlocks blocksize: 87 allocated: 65.3 kb [~ 0%] <~ 768 Blocks blocksize: 88 allocated: 88.0 kb [~ 0%] <~1.00 kBlocks blocksize: 90 allocated: 158 kb [~ 0%] <~1.75 kBlocks blocksize: 91 allocated: 159 kb [~ 0%] <~1.75 kBlocks blocksize: 92 allocated: 322 kb [~ 0%] <~3.50 kBlocks blocksize: 93 allocated: 186 kb [~ 0%] <~2.00 kBlocks blocksize: 95 allocated: 143 kb [~ 0%] <~1.50 kBlocks blocksize: 96 allocated: 240 kb [~ 0%] <~2.50 kBlocks blocksize: 97 allocated: 291 kb [~ 0%] <~3.00 kBlocks blocksize: 98 allocated: 221 kb [~ 0%] <~2.25 kBlocks blocksize: 100 allocated: 75.0 kb [~ 0%] <~ 768 Blocks blocksize: 101 allocated: 50.5 kb [~ 0%] <~ 512 Blocks blocksize: 102 allocated: 51.0 kb [~ 0%] <~ 512 Blocks blocksize: 103 allocated: 77.3 kb [~ 0%] <~ 768 Blocks blocksize: 105 allocated: 158 kb [~ 0%] <~1.50 kBlocks blocksize: 106 allocated: 79.5 kb [~ 0%] <~ 768 Blocks blocksize: 107 allocated: 161 kb [~ 0%] <~1.50 kBlocks blocksize: 108 allocated: 189 kb [~ 0%] <~1.75 kBlocks blocksize: 110 allocated: 110 kb [~ 0%] <~1.00 kBlocks blocksize: 111 allocated: 83.3 kb [~ 0%] <~ 768 Blocks blocksize: 112 allocated: 56.0 kb [~ 0%] <~ 512 Blocks blocksize: 113 allocated: 84.8 kb [~ 0%] <~ 768 Blocks blocksize: 115 allocated: 173 kb [~ 0%] <~1.50 kBlocks blocksize: 116 allocated: 290 kb [~ 0%] <~2.50 kBlocks blocksize: 117 allocated: 263 kb [~ 0%] <~2.25 kBlocks blocksize: 118 allocated: 88.5 kb [~ 0%] <~ 768 Blocks blocksize: 120 allocated: 90.0 kb [~ 0%] <~ 768 Blocks blocksize: 121 allocated: 182 kb [~ 0%] <~1.50 kBlocks blocksize: 122 allocated: 275 kb [~ 0%] <~2.25 kBlocks blocksize: 123 allocated: 246 kb [~ 0%] <~2.00 kBlocks blocksize: 125 allocated: 93.8 kb [~ 0%] <~ 768 Blocks blocksize: 126 allocated: 63.0 kb [~ 0%] <~ 512 Blocks blocksize: 127 allocated: 63.5 kb [~ 0%] <~ 512 Blocks blocksize: 128 allocated: 64.0 kb [~ 0%] <~ 512 Blocks blocksize: 130 allocated: 65.0 kb [~ 0%] <~ 512 Blocks blocksize: 131 allocated: 32.8 kb [~ 0%] <~ 256 Blocks blocksize: 132 allocated: 99.0 kb [~ 0%] <~ 768 Blocks blocksize: 133 allocated: 99.8 kb [~ 0%] <~ 768 Blocks blocksize: 135 allocated: 169 kb [~ 0%] <~1.25 kBlocks blocksize: 136 allocated: 170 kb [~ 0%] <~1.25 kBlocks blocksize: 137 allocated: 103 kb [~ 0%] <~ 768 Blocks blocksize: 138 allocated: 69.0 kb [~ 0%] <~ 512 Blocks blocksize: 140 allocated: 140 kb [~ 0%] <~1.00 kBlocks blocksize: 141 allocated: 70.5 kb [~ 0%] <~ 512 Blocks blocksize: 142 allocated: 107 kb [~ 0%] <~ 768 Blocks blocksize: 143 allocated: 71.5 kb [~ 0%] <~ 512 Blocks blocksize: 145 allocated: 145 kb [~ 0%] <~1.00 kBlocks blocksize: 146 allocated: 256 kb [~ 0%] <~1.75 kBlocks blocksize: 147 allocated: 257 kb [~ 0%] <~1.75 kBlocks blocksize: 148 allocated: 148 kb [~ 0%] <~1.00 kBlocks blocksize: 150 allocated: 37.5 kb [~ 0%] <~ 256 Blocks blocksize: 151 allocated: 75.5 kb [~ 0%] <~ 512 Blocks blocksize: 152 allocated: 152 kb [~ 0%] <~1.00 sum of above: 511 Mb overall alloc: 511 Mb sum of above: 0 b overall alloc: 0 b

epruesse commented 5 years ago

You can remove a lot of those options - there really is no need to re-iterate the defaults. If you want it to be more verbose while it's working, add -v. You should see sequences appearing in the output file though. So in your case something like

tail -f sina_out | grep "^>"

should give you a progress report.

Or, if you have pv installed, try:

pv -d `pgrep sina | tail -n 1`
epruesse commented 5 years ago

Get the 1.5.0 and try this line:

sina -i ~/Sina/amplicons_seeds.fasta -db ~/Sina/sina_ssu.arb -o sina_out.fasta \
   --search --meta-fmt csv --lca-fields tax_slv \
   --fs-engine internal --num-pts 8 -v

you can refine from there.

debjitde01 commented 5 years ago

In online 1.4.0 is the latest version we don't get the 1.5.0. So we tried your script in 1.4.0 but we don't get the output file

(sina) pkd@pkd-HP-406-G1-MT:~/Sina$ sina -i ~/Sina/amplicons_seeds.fasta --db ~/Sina/sina_ssu.arb -o sina_out.fasta --search --meta-fmt csv --lca-fields tax_slv --fs-engine internal --num-pts 8 -v [2019-02-11 11:41:22.044] [log] [info] Loglevel set to info 11:41:22 [SINA] This is SINA 1.4.0. 11:41:22 [libARBDB] ARB: no FastLoad File '/home/pkd/Sina/sina_ssu.ARM' found => loading entire DB 11:41:22 [ARB I/O] Loading names map... (for "/home/pkd/Sina/sina_ssu.arb")

0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----|

11:41:22 [ARB I/O] Loading 38792 sequences...

0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----|

11:41:33 [ARB I/O] Timings for cache load: 10.954 s (1 calls, all = 10.954 s, ) 1.604 s (38792 calls, keys = 1.604 s, ) 9.064 s (38793 calls, arb find = 0.074 s, arb load = 8.990 s, ) 11:41:33 [ARB I/O] Loaded 38792 sequences 11:41:33 [kmer_search] Building index...

0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----|

-------------------- ARB-backtrace 'received signal 11': /home/pkd/anaconda3/envs/sina/bin/../lib/arb/lib/libCORE.so(GBK_dump_backtrace(_IO_FILE, char const)+0x26)[0x7f816d327f36] /home/pkd/anaconda3/envs/sina/bin/../lib/arb/lib/libCORE.so(+0xff34)[0x7f816d329f34] /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20)[0x7f816b753f20] /home/pkd/anaconda3/envs/sina/bin/../lib/libsina.so.0(sina::kmer_search::build_index()+0x403)[0x7f816db7a7d9] /home/pkd/anaconda3/envs/sina/bin/../lib/libsina.so.0(sina::kmer_search::init()+0x18)[0x7f816db7a3d4] /home/pkd/anaconda3/envs/sina/bin/../lib/libsina.so.0(sina::kmer_search::kmer_search(sina::query_arb*, int)+0x66)[0x7f816db7a2fc] /home/pkd/anaconda3/envs/sina/bin/../lib/libsina.so.0(sina::kmer_search::get_kmer_search(boost::filesystem::path&, int)+0x122)[0x7f816db7a0e0] /home/pkd/anaconda3/envs/sina/bin/../lib/libsina.so.0(sina::famfinder::_famfinder::_famfinder(int)+0x175)[0x7f816db00267] /home/pkd/anaconda3/envs/sina/bin/../lib/libsina.so.0(sina::famfinder::finder::finder(int)+0x2c)[0x7f816dafffb4] sina[0x45e918] sina[0x45f2ec] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f816b736b97] sina[0x45bec9] -------------------- End of backtrace [Terminating with signal 11] 11:41:37 [ARB I/O] Closing ARB database '"/home/pkd/Sina/sina_ssu.arb"'

epruesse commented 5 years ago

In online 1.4.0 is the latest version we don't get the 1.5.0. So we tried your script in 1.4.0 but we don't get the output file

If you mean by this that you want to install via conda - the version in bioconda is now updated to 1.5.0, so that should work now.

debjitde01 commented 5 years ago

Now i am trying with the recent version (1.5.0.) in conda and running with your given code, but i am getting an error (6th line from the first line of the script and the 3rd last line). Kindly help me to resolve this.

(ddocent_env) pkd@pkd-HP-406-G1-MT:~/Downloads/sina-1.5.0-linux$ sina -i amplicons_seeds.fasta --db SILVA_132_SSURef_NR99_13_12_17_opt.arb -o sina_out.fasta --outtype fasta --search --meta-fmt csv --lca-fields tax_slv --fs-engine internal --num-pts 8 -v [2019-02-19 14:36:00.523] [log] [info] Loglevel set to info 14:36:00 [SINA] This is SINA 1.5.0. 14:36:00 [libARBDB] ARB: no FastLoad File 'SILVA_132_SSURef_NR99_13_12_17_opt.ARM' found => loading entire DB 14:36:08 [ARB I/O] Loading names map... (for "SILVA_132_SSURef_NR99_13_12_17_opt.arb")

0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----|

14:36:09 [Search (internal)] Index contains 695171 sequences (32088136 refs) 14:36:09 [alignment_stats] alignment stats for subset ssuref:archaea 14:36:09 [alignment_stats] weighted/unweighted columns = 1450/48550 14:36:09 [alignment_stats] average weight = 4.87686 14:36:09 [alignment_stats] minimum weight = 2.13805 14:36:09 [alignment_stats] maximum weight = 7.29296 14:36:09 [alignment_stats] ntaxa = 25025 14:36:09 [alignment_stats] base frequencies: na=0.243357 nu=0.320127 nc=0.238212 ng=0.198304 14:36:09 [alignment_stats] mutation frequencies: any=0.0212039 transversions=0.00822385 14:36:09 [alignment_stats] first/last weighted column=1005/43115 14:36:09 [alignment_stats] alignment stats for subset ssuref:bacteria 14:36:09 [alignment_stats] weighted/unweighted columns = 1532/48468 14:36:09 [alignment_stats] average weight = 5.2707 14:36:09 [alignment_stats] minimum weight = 2.03587 14:36:09 [alignment_stats] maximum weight = 7.67329 14:36:09 [alignment_stats] ntaxa = 592559 14:36:09 [alignment_stats] base frequencies: na=0.252611 nu=0.31454 nc=0.229764 ng=0.203085 14:36:09 [alignment_stats] mutation frequencies: any=0.0167975 transversions=0.00775564 14:36:09 [alignment_stats] first/last weighted column=1006/43241 14:36:09 [alignment_stats] alignment stats for subset ssuref:eukarya 14:36:09 [alignment_stats] weighted/unweighted columns = 1836/48164 14:36:09 [alignment_stats] average weight = 4.74158 14:36:09 [alignment_stats] minimum weight = 2.09547 14:36:09 [alignment_stats] maximum weight = 6.74351 14:36:09 [alignment_stats] ntaxa = 77585 14:36:09 [alignment_stats] base frequencies: na=0.257375 nu=0.268129 nc=0.211923 ng=0.262573 14:36:09 [alignment_stats] mutation frequencies: any=0.0292276 transversions=0.0148656 14:36:09 [alignment_stats] first/last weighted column=1006/43273 14:36:09 [SINA] Launching PT server no 1 14:36:09 [SINA] Launching PT server no 4 14:36:09 [SINA] Launching PT server no 6 14:36:09 [SINA] Launching PT server no 7 14:36:09 [alignment_stats] alignment stats for subset ssuref:archaea 14:36:09 [alignment_stats] weighted/unweighted columns = 1450/48550 14:36:09 [alignment_stats] average weight = 4.87686 14:36:09 [alignment_stats] minimum weight = 2.13805 14:36:09 [alignment_stats] maximum weight = 7.29296 14:36:09 [alignment_stats] ntaxa = 25025 14:36:09 [alignment_stats] base frequencies: na=0.243357 nu=0.320127 nc=0.238212 ng=0.198304 14:36:09 [alignment_stats] mutation frequencies: any=0.0212039 transversions=0.00822385 14:36:09 [alignment_stats] first/last weighted column=1005/43115 14:36:09 [alignment_stats] alignment stats for subset ssuref:bacteria 14:36:09 [alignment_stats] weighted/unweighted columns = 1532/48468 14:36:09 [alignment_stats] average weight = 5.2707 14:36:09 [alignment_stats] minimum weight = 2.03587 14:36:09 [alignment_stats] maximum weight = 7.67329 14:36:09 [alignment_stats] ntaxa = 592559 14:36:09 [alignment_stats] base frequencies: na=0.252611 nu=0.31454 nc=0.229764 ng=0.203085 14:36:09 [alignment_stats] mutation frequencies: any=0.0167975 transversions=0.00775564 14:36:09 [alignment_stats] first/last weighted column=1006/43241 14:36:09 [alignment_stats] alignment stats for subset ssuref:eukarya 14:36:09 [alignment_stats] weighted/unweighted columns = 1836/48164 14:36:09 [alignment_stats] average weight = 4.74158 14:36:09 [alignment_stats] minimum weight = 2.09547 14:36:09 [alignment_stats] maximum weight = 6.74351 14:36:09 [alignment_stats] ntaxa = 77585 14:36:09 [alignment_stats] base frequencies: na=0.257375 nu=0.268129 nc=0.211923 ng=0.262573 14:36:09 [alignment_stats] mutation frequencies: any=0.0292276 transversions=0.0148656 14:36:09 [alignment_stats] first/last weighted column=1006/43273 14:36:09 [SINA] Launching PT server no 2 14:36:09 [alignment_stats] alignment stats for subset ssuref:archaea 14:36:09 [alignment_stats] weighted/unweighted columns = 1450/48550 14:36:09 [alignment_stats] average weight = 4.87686 14:36:09 [alignment_stats] minimum weight = 2.13805 14:36:09 [alignment_stats] maximum weight = 7.29296 14:36:09 [alignment_stats] ntaxa = 25025 14:36:09 [alignment_stats] base frequencies: na=0.243357 nu=0.320127 nc=0.238212 ng=0.198304 14:36:09 [alignment_stats] mutation frequencies: any=0.0212039 transversions=0.00822385 14:36:09 [alignment_stats] first/last weighted column=1005/43115 14:36:09 [alignment_stats] alignment stats for subset ssuref:bacteria 14:36:09 [alignment_stats] weighted/unweighted columns = 1532/48468 14:36:09 [alignment_stats] average weight = 5.2707 14:36:09 [alignment_stats] minimum weight = 2.03587 14:36:09 [alignment_stats] maximum weight = 7.67329 14:36:09 [alignment_stats] ntaxa = 592559 14:36:09 [alignment_stats] base frequencies: na=0.252611 nu=0.31454 nc=0.229764 ng=0.203085 14:36:09 [alignment_stats] mutation frequencies: any=0.0167975 transversions=0.00775564 14:36:09 [alignment_stats] first/last weighted column=1006/43241 14:36:09 [alignment_stats] alignment stats for subset ssuref:eukarya 14:36:09 [alignment_stats] weighted/unweighted columns = 1836/48164 14:36:09 [alignment_stats] average weight = 4.74158 14:36:09 [alignment_stats] minimum weight = 2.09547 14:36:09 [alignment_stats] maximum weight = 6.74351 14:36:09 [alignment_stats] ntaxa = 77585 14:36:09 [alignment_stats] base frequencies: na=0.257375 nu=0.268129 nc=0.211923 ng=0.262573 14:36:09 [alignment_stats] mutation frequencies: any=0.0292276 transversions=0.0148656 14:36:09 [alignment_stats] first/last weighted column=1006/43273 14:36:09 [SINA] Launching PT server no 3 14:36:09 [alignment_stats] alignment stats for subset ssuref:archaea 14:36:09 [alignment_stats] weighted/unweighted columns = 1450/48550 14:36:09 [alignment_stats] average weight = 4.87686 14:36:09 [alignment_stats] minimum weight = 2.13805 14:36:09 [alignment_stats] maximum weight = 7.29296 14:36:09 [alignment_stats] ntaxa = 25025 14:36:09 [alignment_stats] base frequencies: na=0.243357 nu=0.320127 nc=0.238212 ng=0.198304 14:36:09 [alignment_stats] mutation frequencies: any=0.0212039 transversions=0.00822385 14:36:09 [alignment_stats] first/last weighted column=1005/43115 14:36:09 [alignment_stats] alignment stats for subset ssuref:bacteria 14:36:09 [alignment_stats] weighted/unweighted columns = 1532/48468 14:36:09 [alignment_stats] average weight = 5.2707 14:36:09 [alignment_stats] minimum weight = 2.03587 14:36:09 [alignment_stats] maximum weight = 7.67329 14:36:09 [alignment_stats] ntaxa = 592559 14:36:09 [alignment_stats] base frequencies: na=0.252611 nu=0.31454 nc=0.229764 ng=0.203085 14:36:09 [alignment_stats] mutation frequencies: any=0.0167975 transversions=0.00775564 14:36:09 [alignment_stats] first/last weighted column=1006/43241 14:36:09 [alignment_stats] alignment stats for subset ssuref:eukarya 14:36:09 [alignment_stats] weighted/unweighted columns = 1836/48164 14:36:09 [alignment_stats] average weight = 4.74158 14:36:09 [alignment_stats] minimum weight = 2.09547 14:36:09 [alignment_stats] maximum weight = 6.74351 14:36:09 [alignment_stats] ntaxa = 77585 14:36:09 [alignment_stats] base frequencies: na=0.257375 nu=0.268129 nc=0.211923 ng=0.262573 14:36:09 [alignment_stats] mutation frequencies: any=0.0292276 transversions=0.0148656 14:36:09 [alignment_stats] first/last weighted column=1006/43273 14:36:09 [SINA] Launching PT server no 5 14:36:09 [alignment_stats] alignment stats for subset ssuref:archaea 14:36:09 [alignment_stats] weighted/unweighted columns = 1450/48550 14:36:09 [alignment_stats] average weight = 4.87686 14:36:09 [alignment_stats] minimum weight = 2.13805 14:36:09 [alignment_stats] maximum weight = 7.29296 14:36:09 [alignment_stats] ntaxa = 25025 14:36:09 [alignment_stats] base frequencies: na=0.243357 nu=0.320127 nc=0.238212 ng=0.198304 14:36:09 [alignment_stats] mutation frequencies: any=0.0212039 transversions=0.00822385 14:36:09 [alignment_stats] first/last weighted column=1005/43115 14:36:09 [alignment_stats] alignment stats for subset ssuref:bacteria 14:36:09 [alignment_stats] weighted/unweighted columns = 1532/48468 14:36:09 [alignment_stats] average weight = 5.2707 14:36:09 [alignment_stats] minimum weight = 2.03587 14:36:09 [alignment_stats] maximum weight = 7.67329 14:36:09 [alignment_stats] ntaxa = 592559 14:36:09 [alignment_stats] base frequencies: na=0.252611 nu=0.31454 nc=0.229764 ng=0.203085 14:36:09 [alignment_stats] mutation frequencies: any=0.0167975 transversions=0.00775564 14:36:09 [alignment_stats] first/last weighted column=1006/43241 14:36:09 [alignment_stats] alignment stats for subset ssuref:eukarya 14:36:09 [alignment_stats] weighted/unweighted columns = 1836/48164 14:36:09 [alignment_stats] average weight = 4.74158 14:36:09 [alignment_stats] minimum weight = 2.09547 14:36:09 [alignment_stats] maximum weight = 6.74351 14:36:09 [alignment_stats] ntaxa = 77585 14:36:09 [alignment_stats] base frequencies: na=0.257375 nu=0.268129 nc=0.211923 ng=0.262573 14:36:09 [alignment_stats] mutation frequencies: any=0.0292276 transversions=0.0148656 14:36:09 [alignment_stats] first/last weighted column=1006/43273 14:36:09 [alignment_stats] alignment stats for subset ssuref:archaea 14:36:09 [alignment_stats] weighted/unweighted columns = 1450/48550 14:36:09 [alignment_stats] average weight = 4.87686 14:36:09 [alignment_stats] minimum weight = 2.13805 14:36:09 [alignment_stats] maximum weight = 7.29296 14:36:09 [alignment_stats] ntaxa = 25025 14:36:09 [alignment_stats] base frequencies: na=0.243357 nu=0.320127 nc=0.238212 ng=0.198304 14:36:09 [alignment_stats] mutation frequencies: any=0.0212039 transversions=0.00822385 14:36:09 [alignment_stats] first/last weighted column=1005/43115 14:36:09 [alignment_stats] alignment stats for subset ssuref:bacteria 14:36:09 [alignment_stats] weighted/unweighted columns = 1532/48468 14:36:09 [alignment_stats] average weight = 5.2707 14:36:09 [alignment_stats] minimum weight = 2.03587 14:36:09 [alignment_stats] maximum weight = 7.67329 14:36:09 [alignment_stats] ntaxa = 592559 14:36:09 [alignment_stats] base frequencies: na=0.252611 nu=0.31454 nc=0.229764 ng=0.203085 14:36:09 [alignment_stats] mutation frequencies: any=0.0167975 transversions=0.00775564 14:36:09 [alignment_stats] first/last weighted column=1006/43241 14:36:09 [alignment_stats] alignment stats for subset ssuref:eukarya 14:36:09 [alignment_stats] weighted/unweighted columns = 1836/48164 14:36:09 [alignment_stats] average weight = 4.74158 14:36:09 [alignment_stats] minimum weight = 2.09547 14:36:09 [alignment_stats] maximum weight = 6.74351 14:36:09 [alignment_stats] ntaxa = 77585 14:36:09 [alignment_stats] base frequencies: na=0.257375 nu=0.268129 nc=0.211923 ng=0.262573 14:36:09 [alignment_stats] mutation frequencies: any=0.0292276 transversions=0.0148656 14:36:09 [alignment_stats] first/last weighted column=1006/43273 14:36:09 [alignment_stats] alignment stats for subset ssuref:archaea 14:36:09 [alignment_stats] weighted/unweighted columns = 1450/48550 14:36:09 [alignment_stats] average weight = 4.87686 14:36:09 [alignment_stats] minimum weight = 2.13805 14:36:09 [alignment_stats] maximum weight = 7.29296 14:36:09 [alignment_stats] ntaxa = 25025 14:36:09 [alignment_stats] base frequencies: na=0.243357 nu=0.320127 nc=0.238212 ng=0.198304 14:36:09 [alignment_stats] mutation frequencies: any=0.0212039 transversions=0.00822385 14:36:09 [alignment_stats] first/last weighted column=1005/43115 14:36:09 [alignment_stats] alignment stats for subset ssuref:bacteria 14:36:09 [alignment_stats] weighted/unweighted columns = 1532/48468 14:36:09 [alignment_stats] average weight = 5.2707 14:36:09 [alignment_stats] minimum weight = 2.03587 14:36:09 [alignment_stats] maximum weight = 7.67329 14:36:09 [alignment_stats] ntaxa = 592559 14:36:09 [alignment_stats] base frequencies: na=0.252611 nu=0.31454 nc=0.229764 ng=0.203085 14:36:09 [alignment_stats] mutation frequencies: any=0.0167975 transversions=0.00775564 14:36:09 [alignment_stats] first/last weighted column=1006/43241 14:36:09 [alignment_stats] alignment stats for subset ssuref:eukarya 14:36:09 [alignment_stats] weighted/unweighted columns = 1836/48164 14:36:09 [alignment_stats] average weight = 4.74158 14:36:09 [alignment_stats] minimum weight = 2.09547 14:36:09 [alignment_stats] maximum weight = 6.74351 14:36:09 [alignment_stats] ntaxa = 77585 14:36:09 [alignment_stats] base frequencies: na=0.257375 nu=0.268129 nc=0.211923 ng=0.262573 14:36:09 [alignment_stats] mutation frequencies: any=0.0292276 transversions=0.0148656 14:36:09 [alignment_stats] first/last weighted column=1006/43273 14:36:09 [alignment_stats] alignment stats for subset ssuref:archaea 14:36:09 [alignment_stats] weighted/unweighted columns = 1450/48550 14:36:09 [alignment_stats] average weight = 4.87686 14:36:09 [alignment_stats] minimum weight = 2.13805 14:36:09 [alignment_stats] maximum weight = 7.29296 14:36:09 [alignment_stats] ntaxa = 25025 14:36:09 [alignment_stats] base frequencies: na=0.243357 nu=0.320127 nc=0.238212 ng=0.198304 14:36:09 [alignment_stats] mutation frequencies: any=0.0212039 transversions=0.00822385 14:36:09 [alignment_stats] first/last weighted column=1005/43115 14:36:09 [alignment_stats] alignment stats for subset ssuref:bacteria 14:36:09 [alignment_stats] weighted/unweighted columns = 1532/48468 14:36:09 [alignment_stats] average weight = 5.2707 14:36:09 [alignment_stats] minimum weight = 2.03587 14:36:09 [alignment_stats] maximum weight = 7.67329 14:36:09 [alignment_stats] ntaxa = 592559 14:36:09 [alignment_stats] base frequencies: na=0.252611 nu=0.31454 nc=0.229764 ng=0.203085 14:36:09 [alignment_stats] mutation frequencies: any=0.0167975 transversions=0.00775564 14:36:09 [alignment_stats] first/last weighted column=1006/43241 14:36:09 [alignment_stats] alignment stats for subset ssuref:eukarya 14:36:09 [alignment_stats] weighted/unweighted columns = 1836/48164 14:36:09 [alignment_stats] average weight = 4.74158 14:36:09 [alignment_stats] minimum weight = 2.09547 14:36:09 [alignment_stats] maximum weight = 6.74351 14:36:09 [alignment_stats] ntaxa = 77585 14:36:09 [alignment_stats] base frequencies: na=0.257375 nu=0.268129 nc=0.211923 ng=0.262573 14:36:09 [alignment_stats] mutation frequencies: any=0.0292276 transversions=0.0148656 14:36:09 [alignment_stats] first/last weighted column=1006/43273 14:36:09 [Search (ARB PT)] Using ARBHOME="/home/pkd/anaconda3/envs/ddocent_env/lib/arb" 14:36:09 [Search (ARB PT)] PT server index out of date for SILVA_132_SSURef_NR99_13_12_17_opt.arb. Rebuilding: ARB: Loading 'SILVA_132_SSURef_NR99_13_12_17_opt.arb.index.arb' ARB: no FastLoad File 'SILVA_132_SSURef_NR99_13_12_17_opt.arb.index.ARM' found => loading entire DB ARB: Loading 'SILVA_132_SSURef_NR99_13_12_17_opt.arb.index.arb' done

Progress: Remove unused database entries ...................................................................... [ 8.3%] left: 1m49s ...................................................................... [ 16.7%] left: 1m39s ...................................................................... [ 25.0%] left: 1m27s ...................................................................... [ 33.3%] left: 1m18s ...................................................................... [ 41.7%] left: 1m8s ...................................................................... [ 50.0%] left: 58s ...................................................................... [ 58.3%] left: 48s ...................................................................... [ 66.7%] left: 39s ...................................................................... [ 75.0%] left: 30s ...................................................................... [ 83.3%] left: 20s ...................................................................... [ 91.7%] left: 10s ...................................................................... [100.0%] used: 1m57s [done] Database contains 695171 species Progress: Preparing sequence data ...................................................................... [ 8.3%] left: 1m38s ...................................................................... [ 16.7%] left: 1m34s ...................................................................... [ 25.0%] left: 1m24s ...................................................................... [ 33.3%] left: 1m14s ...................................................................... [ 41.7%] left: 1m4s ...................................................................... [ 50.0%] left: 54s ...................................................................... [ 58.3%] left: 45s ...................................................................... [ 66.7%] left: 36s ...................................................................... [ 75.0%] left: 28s ...................................................................... [ 83.3%] left: 19s ...................................................................... [ 91.7%] left: 9s ...................................................................... [100.0%] used: 1m55s [done]

All species contain data in alignment 'ali_16s'. [ptserver '-build_clean' took 6m41s] sum of above: 0 b overall alloc: 0 b sum of above: 0 b overall alloc: 0 b ARB: Loading 'SILVA_132_SSURef_NR99_13_12_17_opt.arb.index.arb' ARB: Opening FastLoad File 'SILVA_132_SSURef_NR99_13_12_17_opt.arb.index.ARM' ... ARB: Loading 'SILVA_132_SSURef_NR99_13_12_17_opt.arb.index.arb' done

Building PT-Server for alignment 'ali_16s'... Database contains 695171 species Progress: Checking data ...................................................................... [ 8.3%] left: 1m16s ...................................................................... [ 16.7%] left: 40s ...................................................................... [ 25.0%] left: 27s ...................................................................... [ 33.3%] left: 18s ...................................................................... [ 41.7%] left: 14s ...................................................................... [ 50.0%] left: 11s ...................................................................... [ 58.3%] left: 9s ...................................................................... [ 66.7%] left: 6s ...................................................................... [ 75.0%] left: 4s ...................................................................... [ 83.3%] left: 3s ...................................................................... [ 91.7%] left: 1s ...................................................................... [100.0%] used: 15s [done] Failed to write to '/home/pkd/anaconda3/envs/ddocent_env/lib/arb/lib/pts/ptserver.log' Visible memory: 11.7 Gb Restricting used memory (by internal default '90%') to 10.5 Gb Note: Setting envar ARB_MEMORY will override that restriction (percentage or absolute memsize) Memory available for build: 9.21 Gb Estimated memory usage for 1 pass: 6.02 Gb Overall bases: 964 Mbp Max. partition size: 964 Mbp (=100.0%) Progress: Build index in 1 passes {pass 1/1}............................................................ [ 8.3%] left: 15m16s ...................................................................... [ 16.7%] left: 15m36s ...................................................................... [ 25.0%] left: 14m36s ...................................................................... [ 33.3%] left: 13m10s ...................................................................... [ 41.7%] left: 11m40s ...................................................................... [ 50.0%] left: 10m6s ...................................................................... [ 58.3%] left: 8m30s ...................................................................... [ 66.7%] left: 6m53s ...................................................................... [ 75.0%] left: 5m17s ..........................................................Killed 05:57:19 [FASTA I/O] read -1 sequences from -1 lines 05:57:25 [SINA] Error during program execution: Failed to (re)build PT server index! (out of memory?) 05:57:25 [ARB I/O] Closing ARB database '"SILVA_132_SSURef_NR99_13_12_17_opt.arb"' ... 06:02:54 [Search (internal)] Timings for Kmer Search: never called

epruesse commented 5 years ago

This is what the PT server said:

Visible memory: 11.7 Gb

Ok, so you have some 12GB of RAM in your computer.

Restricting used memory (by internal default '90%') to 10.5 Gb

ARB thinks it's ok to use 90% of that, so 10.5 GB

Note: Setting envar ARB_MEMORY will override that restriction (percentage or absolute memsize)

ARB tells you to set ARB_MEMORY if you need this to be lower.

Memory available for build: 9.21 Gb Estimated memory usage for 1 pass: 6.02 Gb Overall bases: 964 Mbp Max. partition size: 964 Mbp (=100.0%) Progress: Build index in 1 passes {pass 1/1}............................................................ [ 8.3%] left: 15m16s ...................................................................... [ 16.7%] left: 15m36s ...................................................................... [ 25.0%] left: 14m36s ...................................................................... [ 33.3%] left: 13m10s ...................................................................... [ 41.7%] left: 11m40s ...................................................................... [ 50.0%] left: 10m6s ...................................................................... [ 58.3%] left: 8m30s ...................................................................... [ 66.7%] left: 6m53s ...................................................................... [ 75.0%] left: 5m17s ..........................................................Killed

ARB dies.

05:57:19 [FASTA I/O] read -1 sequences from -1 lines 05:57:25 [SINA] Error during program execution: Failed to (re)build PT server index! (out of memory?) 05:57:25 [ARB I/O] Closing ARB database '"SILVA_132_SSURef_NR99_13_12_17_opt.arb"' ... 06:02:54 [Search (internal)] Timings for Kmer Search: never called

SINA can't go on.

So basically, try this:ARB_MEMORY=70% sina ...

epruesse commented 5 years ago

I'm guessing you found a way to make this work. It was probably an out of memory condition in the PT server. The upcoming 1.6 will default to the internal search engine which uses a lot less memory.

debjitde01 commented 5 years ago

Is the Sina 1.6.0 available or not? and how can I update the Sina 1.5.0 to 1.6.0, please help on this issue..

Thankinf you with regards

Debjit De

Research Scholar C/O - Dr. Paltu Kumar Dhal Assistant Professor Department of Life Science & Biotechnology Jadavpur University Kolkata - 700032.

On Sun, Mar 24, 2019 at 12:15 AM Elmar Pruesse notifications@github.com wrote:

I'm guessing you found a way to make this work. It was probably an out of memory condition in the PT server. The upcoming 1.6 will default to the internal search engine which uses a lot less memory.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/epruesse/SINA/issues/53#issuecomment-475894710, or mute the thread https://github.com/notifications/unsubscribe-auth/AbX7r6ZIFtiSFgtVaGkRwbtEpw_XdPLCks5vZnZUgaJpZM4ayfP7 .

epruesse commented 5 years ago

It is not yet on Bioconda, but it is available as a direct download.

Just download one of these: linux or macos, unpack with tar xf sina-1.6.0-rc.1*.tar.gz and then run the sina that is in the folder created by tar (e.g. ./sina-1.6.0*/sina --help).

It's still release candidate - that is while my tests have concluded, there may yet be issues that my use won't show. If you can give it a spin, I'd appreciate that greatly! Perhaps it resolves your problems. If not, we can debug them and add necessary fixes to the final 1.6.0.