search

galaxyproject / tools-iuc

Tool Shed repositories maintained by the Intergalactic Utilities Commission
https://galaxyproject.org/iuc
MIT License
160 stars 417 forks source link

mothur.cluster.split hangs compute nodes due to an uncaught error #3663

Open gmauro opened 3 years ago

gmauro commented 3 years ago

I found several compute nodes stuck with these kinds of message in the syslog:

[Wed May 12 10:23:20 2021] mothur[890441]: segfault at 2047c70116 ip 0000000000806cf8 sp 00007ffed82d5870 error 4 in mothur[400000+eab000]
[Wed May 12 10:23:20 2021] mothur[890456]: segfault at 168 ip 0000000000806cf8 sp 00007ffed82d5870 error 4 in mothur[400000+eab000]
[Wed May 12 10:23:20 2021] Code: 66 90 48 85 db 74 73 48 8d 53 08 c7 43 08 00 00 00 00 48 c7 43 10 00 00 00 00 48 c7 43 28 00 00 00 00 48 89 53 18 48 89 53 20 <48> 8b 75 10 48 85 f6 74 47 48 89 df e8 c7 20 c3 ff
 48 89 43 10 48
[Wed May 12 10:23:20 2021] Code: 66 90 48 85 db 74 73 48 8d 53 08 c7 43 08 00 00 00 00 48 c7 43 10 00 00 00 00 48 c7 43 28 00 00 00 00 48 89 53 18 48 89 53 20 <48> 8b 75 10 48 85 f6 74 47 48 89 df e8 c7 20 c3 ff
 48 89 43 10 48
[Wed May 12 10:23:20 2021] SLUB: Unable to allocate memory on node -1, gfp=0x6000c0(GFP_KERNEL)
[Wed May 12 10:23:20 2021]   cache: task_struct(18113:condor_var_lib_condor_execute_slot1_5@vgcnbwc-worker-c125m425-2069.novalocal), object size: 6080, buffer size: 6080, default order: 3, min order: 1
[Wed May 12 10:23:20 2021]   node 0: slabs: 85995, objs: 429743, free: 0

In all nodes was running mothur.cluster.split 1.39.5 and this is its sdtout:

mothur v.1.39.5
Last updated: 3/20/2017

by
Patrick D. Schloss

Department of Microbiology & Immunology
University of Michigan
http://www.mothur.org

When using, please cite:
Schloss, P.D., et al., Introducing mothur: Open-source, platform-independent, community-supported software for describing and comparing microbial communities. Appl Environ Microbiol, 2009. 75(23):7537-41.

Distributed under the GNU General Public License

Type 'help()' for information on the commands that are available

For questions and analysis support, please visit our forum at https://www.mothur.org/forum

Type 'quit()' to exit program

se,cluster=true,runsensspec=true,processors=8),iters=100,precision=100,large=fal 

Using 8 processors.
Splitting the file...
/******************************************/
Running command: dist.seqs(fasta=splitby.fasta.dat.0.temp, processors=8, cutoff=0.03)

Using 8 processors.
/******************************************/
[ERROR]: your sequences are not the same length, aborting.
It took 1 seconds to split the distance file.
[ERROR]: std::bad_allocRAM used: 0.00388718Gigabytes . Total Ram: 211.266Gigabytes.

 has occurred in the ClusterSplitCommand class function createProcesses. This error indicates your computer is running out of memory.  This is most commonly caused by trying to process a dataset too large, using
 multiple processors, or a file format issue. If you are running our 32bit version, your memory usage is limited to 4G.  If you have more than 4G of RAM and are running a 64bit OS, using our 64bit version may re
solve your issue.  If you are using multiple processors, try running the command with processors=1, the more processors you use the more memory is required. Also, you may be able to reduce the size of your datas
et by using the commands outlined in the Schloss SOP, http://www.mothur.org/wiki/Schloss_SOP. If you are unable to resolve the issue, please contact Pat Schloss at mothur.bugs@gmail.com, and be sure to include t
he mothur.logFile with your inquiry.[ERROR]: std::bad_allocRAM used: 0.00382233Gigabytes . Total Ram: 211.266Gigabytes.

and this is an extract from the fasta file:

>HE855366.1.<1.>570
TCTGTGCTTATTCGTATGGAATTAGCTGGTCCGGGAGTTCAAGTTTTAGGTGGAAATCAT
CAATTATATAATGTTATAGTTACAGGTCACGCTTTTATAATGATTTTTTTTATGGTTATG
CCTGTTCTAATGGGTGGTTATGGTAATTGGTTTGTTCCTATTATGATAGGAGCTCCTGAT
ATGGCTTTTCCTAGAATGAATAATATAAGTTTTTGGTTATTACCACCTTCTTTAATTTTG
TTATTGAGTTCTACATTGGTAGAAATAGGTGTTGGTACTGGTTGGACCGTGTATCCTCCG
TTAAGTAGTATCTCTGGACATCCTGGAGGCGCAGTTGATTTAGGTATATTTAGTTTGCAT
GTAGCAGGTGCTTCCAGTATCTTAGGCGCTATTAATTTTATAACAACTATTTTTAATATG
AGAGTGCCAGGCATGACAATGCACAGAATACCTCTCTTTGTTTGGGCTGTTTTAATAACT
GCTTTTTTACTTCTTTTATCATTACCTGTTTTTGCTGGTGCTATTACTATGTTATTAACT
GATAGAAATTTTAATACAAGTTTTTTTGAC
>AB009419.1.<1.>1059
GGTACATTATATATACTTTTTGGAATAATATCAGGTATTATAGGTACAACTTTGTCTGTT
CTTATAAGGATGGAATTAGCAGGCCCAGGTGTCCAAGTTTTAGGAGGTAATCACCAATTA
TATAACGTTATTATTACTGGTCATGCTTTTATAATGATATTTTTCATGGTAATGCCAGTA
TTGATTGGAGGTTACGGTAACTGGTTTATTCCTATTATGATAGGTGCACCTGATATGAGT
TTCCCAAGAATGAATAATATAAGCTTTTGGTTACTACCACCATCTTTAATTTTGTTATTA
AGCTCTACTCTTGTTGAAGTTGGTGTTGGCACTGGCTGGACAGTATACCCTCCCTTAAGC
TCTATCTCTGGTCATCCCGGTGCAGCAGTTGATTTAGGTATTTTCAGTCTTCATATTGCA
GGTGCATCTAGTATTTTAGGTTCTATTAATTTTATAACTACTATTTTTAATATGAGAGGT
CCTGGAATGACTATGCATAGAATACCTTTGTTTGTTTGGGCTGTTCTTATAACTGCTTTT
TTACTTGTGCTGTCACTTCCTGTATTTGCTGGGGCAATCACTATGCTTCTAACCGATCGT
AATTTTAATACAAGCTTTTTTGAAGCAGCAGGTGGTGGAGACCCTGTTTTATACCAACAT
TTGTTTTGGTTTTTTGGTCATCCGGAAGTTTATATCTTGATATTACCTGCTTTCGGAATT
ATTAGTCATATAACTTCTACTTTTTCTAGAAAACCAGTTTTTGGTTTTATAGGTATGGTA
TATGCCATGTTGAGTATAGGTCTATTAGGTTTTATTGTTTGGGCACATCATATGTATACT
GTCGGTATGGATATTGACACTAGAGCTTACTTTACAGCAGCTACTATGATTATAGCTGTC
CCAACTGGAATAAAAATATTTAGTTGGATAGCGACTATGTGGGGAGGCTCTATTTATCTA
AAAACCCCAATGGTTTTTGCTTTAGGCTTTATATGCTTGTTTACAATTGGAGGGTTATCT
GGTATTATGTTATCGAATGGTGCTTTGGATATAGCCTTT
>EU651892.1.933.2531
ATGTCAAATTTTTTAAATCGCTGGATTTTTTCGACAAATCATAAAGATATTGGTACATTA
TATCTAATTTTTGCAATTTTTGCGGGAGTTGTAGGTACTTTTTTATCGGTTTTAATTCGA
TTGGAATTAGCTGGGCCTGGCGTTCAAATATTAGGGGGTAACCACCAATTATATAACGTA
ATTATTACAGCTCATGCCTTTGTGATGATTTTTTTTATGGTGATGCCTGCACTAATTGGA
GGTTTCGCAAACTGGTTTGTTCCTATTATGATAGGTGCTCCAGATATGGCTTTTCCTCGT
TTAAATAACATTTCTTTCTGGTTATTAATACCTGCCTTCGTTTTATTATTAAGTTCATCA
TTCGTAGAAACTGGTGCGGGTACTGGCTGGACAGTGTACCCACCGTTAAGTAGTATAAGT
GGGCACCCTGGTGGATCTGTGGATTTAGCTATATTTAGCCTTCACGTTGCAGGGGCCTCA
AGTATTTTAGGTGCTTGTAATTTTATTACAACAATTCTTAATATGCGAGCACCAGGGATG
ACATTACACCGATTGCCACTTTTTTGCTGGGCAGTATTAATTACTGCGGTTTTATTAGTA
CTATCACTACCAGTATTTGCAGGGGCGATAACGATGTTGCTTACAGATAGAAATTTTAAT
ACGGCATTTTTTGATGCTAGTCTTGGCGGTGACCCAGTTCTTTATGAACATCTTTTTTGG
TTCTTTGGGCATCCTGAAGTTTATATATTAATTTTACCTGGATTTGGTATTATCAGTCAC
ATCGTATCTACTTTTTCAAGAAAACCTGTTTTCGGTGTAATTGGTATGATTTATGCAATG
GTTAGTATTGGTGTTCTCGGCTTTATAGTGTGGGCGCACCATATGTACACCGTTGGAATG
GACGTAACAACAAGAGCTTATTTCACAGCAACAACAATGGTAATTGCAGTACCTACCGGT
ATCAAGATTTTCTCGTGGATTGCTACAATGTGGGGGGGTTCAATTCATTTGAAAACACCA
ATGGTTTTTGCTATTGGTTTCATTTTCTTATTCACAATTGGTGGATTAACAGGAGTGGTT
CTTTCAAATGGTGGTTTAGATTTAGCGTTCCATGACAGTTATTACGTTGTGGCACATTTT
CACTATGTTCTCTCTATGGGAGCAGTATTCTCAATGTTTGCTGGTTACTATTATTGGATT
GGAAAAATGTCAGGATTTAATTATCCAGAAAATCTCGGAATTATTCACTTTTGGTGCACT
TTTGTAGGGGTTAATTGTACTTTCTTTCCACAGCACTTTTTAGGTTTAGCAGGGATGCCA
AGAAGAATACCAGATTATCCTGATGCATATGCAGGTTGGAATTATATTTCATCTTTTGGT
AGTTCAATTTCGGTTTTTGCAATTCTTTTATTTTTTGTTTTGACGTACGAAACATTTACA
AATATGGACAAATGTCCGGTAAATCCTTGGTCGTTTGCAACATCAAGTGCTGATCCAAAA
TTTGAATTTACGCTTGAATGGGTTGTAGGTTCACCACCATCTTTTCATACTTTTGAGGAA
TTACCTATTATCAAAGATACTGATATAGTAAATGTGTAG

From the point of view of Galaxy and of Condor, the job is still running, but instead it's badly crashed

gmauro commented 3 years ago

ping @bebatut @shiltemann

gmauro commented 3 years ago

reducing processors to 1, the behavior is the same: mothur crashes but Galaxy ignore it (and the load in the node ramp up to infinite :\ 

mothur v.1.39.5
Last updated: 3/20/2017

by
Patrick D. Schloss

Department of Microbiology & Immunology
University of Michigan
http://www.mothur.org

When using, please cite:
Schloss, P.D., et al., Introducing mothur: Open-source, platform-independent, community-supported software for describing and comparing microbial communities. Appl Environ Microbiol, 2009. 75(23):7537-41.

Distributed under the GNU General Public License

Type 'help()' for information on the commands that are available

For questions and analysis support, please visit our forum at https://www.mothur.org/forum

Type 'quit()' to exit program

ors=1)utoff=0.15,precision=100,large=false,cluster=true,runsensspec=true,process 

Using 1 processors.
Splitting the file...
/******************************************/
Running command: dist.seqs(fasta=splitby.fasta.dat.0.temp, processors=1, cutoff=0.15)

Using 1 processors.
/******************************************/
[ERROR]: your sequences are not the same length, aborting.
It took 0 seconds to split the distance file.
bernt-matthias commented 3 years ago

I guess the relevant error is [ERROR]: your sequences are not the same length, aborting.

Probably you need aligned data in the fasta?

bgruening commented 3 years ago

Probably, but our problem is that the tool does not abort and still runs and ruins the node somehow.

bernt-matthias commented 3 years ago

and the load in the node ramp up to infinite

what does this mean?

In both cases Galaxy should be able to detect the error after the tool finished: https://github.com/galaxyproject/tools-iuc/blob/9a163dd8880c14f371e2603389f4951881a74b25/tools/mothur/macros.xml#L14

So I guess the tool does not finish. If so, then there is not much that the Galaxy tool can do about it, but its an upstream problem.

gmauro commented 3 years ago

the system load is 10000x over a "normal" value

gmauro commented 3 years ago

Today I found the same situation with a different mothur tool. Probably should be upgraded to the last version

gmauro commented 3 years ago 
top - 16:34:53 up 22:10,  1 user,  load average: 19391.86, 19368.43, 17192.83
Tasks: 25727 total, 157 running, 20337 sleeping,   0 stopped, 5233 zombie
%Cpu(s):  2.1 us,  7.0 sy,  0.0 ni, 90.8 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 428015.6 total,  35536.1 free,  13283.9 used, 379195.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 411225.3 avail Mem 
centos@vgcnbwc-worker-c125m425-9515:~$ ps aux | grep mothur|wc -l
24621
hexylena commented 2 years ago

@shiltemann

Slugger70 commented 1 year ago

@hexylena, @bgruening, @bernt-matthias, @shiltemann - I think I've found part of the problem. When Mothur cluster.split encounters something unusual it gets itself stuck in a loop forking off child processes that all want to talk to the disk. I managed to see it breaking one of my VMs last week and it had created > 20k child processes and was still creating them. I attempted to stop it with pkill -9 mothur to no avail. It kept creating processes. I ended up having to kill the VM to stop it. The version of Mothur that we have in Galaxy is quite old now. We currently run 1.39.5 (2018) - the current version is 1.48.0 and there have been a number of bug fixes for cluster.split.

Do we have any plans for updating these tools? I understand that any update is a huge amount of work and am not sure there is anyone willing to do it.

At the moment I don't know what triggers the current bad behaviour so I can't look for it before the tool starts running. In the meantime, I've disabled the tool on Galaxy Australia as it keeps bringing down compute nodes.

shiltemann commented 1 year ago

Hey @Slugger70, thanks for the update, yikes. yes, updating the whole suite would be a lot of work, but we don't have to do everything at once and starting with just cluster.split might not be so much work, I can look into it

shiltemann commented 1 year ago

Today I found the same situation with a different mothur tool. Probably should be upgraded to the last version

@gmauro do you remember which other mothur tool you noticed this bad behaviour with?

bernt-matthias commented 1 year ago

Hey @shiltemann .. Seems like a good idea.

Wondering if updating a single tool is a good idea, since we do not know if it plays well with older versions of other mothur tools. Maybe the usegalaxy's can provide some usage stats, such that the most frequently used tools can be updated?

shiltemann commented 1 year ago

I agree it's not ideal, but I guess better than no tool at all (since AU was forced to disable the tool altogether now). Guess it all depends on how much has changed in the intervening versions, which I haven't looked into in detail yet.

My guess would be that all the tools used in the tutorial are the most-used ones, since they comprise the SOP (but if we have actual usage data I would be curious to see if this is correct), so getting that subset of tools updated would be our priority, or at least make sure that different versions of tools play nicely together in the SOP case

bgruening commented 1 year ago 
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_dist_seqs/mothur_dist_seqs/1.39.5.0                                                                         |   20235
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_seqs/mothur_summary_seqs/1.39.5.0                                                                   |   20211
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_contigs/mothur_make_contigs/1.39.5.1                                                                   |   14627
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_unique_seqs/mothur_unique_seqs/1.39.5.0                                                                     |   14203
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_taxonomy_to_krona/mothur_taxonomy_to_krona/1.0                                                              |   13167
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_count_seqs/mothur_count_seqs/1.39.5.0                                                                       |    8820
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_align_seqs/mothur_align_seqs/1.39.5.0                                                                       |    8804
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_contigs/mothur_make_contigs/1.39.5.0                                                                   |    6994
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_screen_seqs/mothur_screen_seqs/1.39.5.0                                                                     |    6899
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_screen_seqs/mothur_screen_seqs/1.39.5.1                                                                     |    6533
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_classify_seqs/mothur_classify_seqs/1.39.5.0                                                                 |    5961
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_classify_otu/mothur_classify_otu/1.39.5.0                                                                   |    5421
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_cluster_split/mothur_cluster_split/1.39.5.0                                                                 |    5385
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_pre_cluster/mothur_pre_cluster/1.39.5.0                                                                     |    4612
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_filter_seqs/mothur_filter_seqs/1.39.5.0                                                                     |    4598
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_shared/mothur_make_shared/1.39.5.0                                                                     |    4433
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_count_groups/mothur_count_groups/1.39.5.0                                                                   |    4378
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_heatmap_sim/mothur_heatmap_sim/1.39.5.0                                                                     |    4137
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_biom/mothur_make_biom/1.39.5.0                                                                         |    4100
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_vsearch/mothur_chimera_vsearch/1.39.5.1                                                             |    4075
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_fastq_info/mothur_fastq_info/1.39.5.0                                                                       |    3807
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_tree_shared/mothur_tree_shared/1.39.5.0                                                                     |    3796
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_lineage/mothur_remove_lineage/1.39.5.0                                                               |    3505
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_group/mothur_make_group/1.39.5.0                                                                       |    3196
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_seqs/mothur_remove_seqs/1.39.5.0                                                                     |    2819
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_rarefaction_single/mothur_rarefaction_single/1.39.5.0                                                       |    2675
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_sub_sample/mothur_sub_sample/1.39.5.0                                                                       |    2655
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_files/mothur_merge_files/1.39.5.0                                                                     |    2431
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_seqs/mothur_summary_seqs/1.36.1.0                                                                   |    2391
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_trim_seqs/mothur_trim_seqs/1.39.5.0                                                                         |    1952
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_dist_shared/mothur_dist_shared/1.39.5.0                                                                     |    1899
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_single/mothur_summary_single/1.39.5.0                                                               |    1443
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_venn/mothur_venn/1.39.5.0                                                                                   |    1208
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_seqs/mothur_remove_seqs/1.36.1.0                                                                     |    1197
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_groups/mothur_remove_groups/1.39.5.0                                                                 |    1116
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_cluster/mothur_cluster/1.39.5.0                                                                             |     990
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_groups/mothur_get_groups/1.39.5.0                                                                       |     948
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_contigs/mothur_make_contigs/1.36.1.0                                                                   |     938
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_tax/mothur_summary_tax/1.39.5.0                                                                     |     869
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_biom/mothur_make_biom/1.36.1.0                                                                         |     826
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_fastq/mothur_make_fastq/1.36.1.0                                                                       |     771
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_single/mothur_summary_single/1.39.5.2                                                               |     727
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_rare/mothur_remove_rare/1.39.5.0                                                                     |     689
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_pcoa/mothur_pcoa/1.39.5.0                                                                                   |     673
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_biom_info/mothur_biom_info/1.39.5.0                                                                         |     643
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_uchime/mothur_chimera_uchime/1.39.5.0                                                               |     599
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_seq_error/mothur_seq_error/1.39.5.0                                                                         |     569
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_count_seqs/mothur_count_seqs/1.36.1.0                                                                       |     482
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_screen_seqs/mothur_screen_seqs/1.36.1.0                                                                     |     472
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_slayer/mothur_chimera_slayer/1.39.5.0                                                               |     469
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_reverse_seqs/mothur_reverse_seqs/1.39.5.0                                                                   |     406
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_phylotype/mothur_phylotype/1.39.5.0                                                                         |     397
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_oturep/mothur_get_oturep/1.39.5.0                                                                       |     395
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_coremicrobiome/mothur_get_coremicrobiome/1.39.5.0                                                       |     375
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_rename_seqs/mothur_rename_seqs/1.39.5.0                                                                     |     373
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_shared/mothur_make_shared/1.36.1.0                                                                     |     351
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_heatmap_bin/mothur_heatmap_bin/1.39.5.0                                                                     |     344
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_pcr_seqs/mothur_pcr_seqs/1.39.5.0                                                                           |     307
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_relabund/mothur_get_relabund/1.39.5.0                                                                   |     302
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_amova/mothur_amova/1.39.5.0                                                                                 |     291
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_unique_seqs/mothur_unique_seqs/1.36.1.0                                                                     |     280
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_lineage/mothur_remove_lineage/1.36.1.0                                                               |     257
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_groups/mothur_merge_groups/1.39.5.0                                                                   |     254
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_align_seqs/mothur_align_seqs/1.36.1.0                                                                       |     252
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_pre_cluster/mothur_pre_cluster/1.36.1.0                                                                     |     237
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_classify_seqs/mothur_classify_seqs/1.36.1.0                                                                 |     233
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_cluster_split/mothur_cluster_split/1.36.1.0                                                                 |     231
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_classify_otu/mothur_classify_otu/1.36.1.0                                                                   |     228
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_lineage/mothur_get_lineage/1.39.5.0                                                                     |     222
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_group/mothur_make_group/1.36.1.0                                                                       |     222
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_list_seqs/mothur_list_seqs/1.39.5.0                                                                         |     221
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_filter_seqs/mothur_filter_seqs/1.36.1.0                                                                     |     216
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_shared/mothur_summary_shared/1.39.5.0                                                               |     211
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_files/mothur_merge_files/1.36.1.0                                                                     |     211
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_sort_seqs/mothur_sort_seqs/1.39.5.0                                                                         |     199
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_heatmap_sim/mothur_heatmap_sim/1.36.1.0                                                                     |     194
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_tree_shared/mothur_tree_shared/1.36.1.0                                                                     |     185
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_lefse/mothur_make_lefse/1.39.5.0                                                                       |     176
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_unifrac_unweighted/mothur_unifrac_unweighted/1.39.5.0                                                       |     176
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_parsimony/mothur_parsimony/1.39.5.0                                                                         |     175
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_unifrac_weighted/mothur_unifrac_weighted/1.39.5.0                                                           |     174
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_qual/mothur_summary_qual/1.39.5.0                                                                   |     174
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_vsearch/mothur_chimera_vsearch/1.39.5.0                                                             |     173
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_lefse/mothur_lefse/1.39.5.0                                                                                 |     170
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_split_abund/mothur_split_abund/1.39.5.0                                                                     |     167
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_homova/mothur_homova/1.39.5.0                                                                               |     167
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_sharedseqs/mothur_get_sharedseqs/1.39.5.0                                                               |     159
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_clearcut/mothur_clearcut/1.39.5.0                                                                           |     156
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_seqs/mothur_get_seqs/1.39.5.0                                                                           |     155
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_rarefaction_single/mothur_rarefaction_single/1.36.1.0                                                       |     154
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_nmds/mothur_nmds/1.39.5.0                                                                                   |     152
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_metastats/mothur_metastats/1.39.5.0                                                                         |     139
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_indicator/mothur_indicator/1.39.5.0                                                                         |     126
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_corr_axes/mothur_corr_axes/1.39.5.0                                                                         |     125
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_tax/mothur_summary_tax/1.36.1.0                                                                     |     118
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_single/mothur_summary_single/1.36.1.0                                                               |     113
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_deunique_seqs/mothur_deunique_seqs/1.39.5.0                                                                 |     111
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_venn/mothur_venn/1.36.1.0                                                                                   |     109
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_label/mothur_get_label/1.39.5.0                                                                         |     108
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_degap_seqs/mothur_degap_seqs/1.39.5.0                                                                       |     100
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_fastq/mothur_make_fastq/1.39.5.0                                                                       |     100
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_rarefaction_shared/mothur_rarefaction_shared/1.39.5.0                                                       |      98
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_pairwise_seqs/mothur_pairwise_seqs/1.39.5.0                                                                 |      91
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_split_groups/mothur_split_groups/1.39.5.0                                                                   |      90
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_create_database/mothur_create_database/1.39.5.0                                                             |      88
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_filter_shared/mothur_filter_shared/1.39.5.0                                                                 |      87
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chop_seqs/mothur_chop_seqs/1.39.5.0                                                                         |      87
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_pca/mothur_pca/1.39.5.0                                                                                     |      87
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_count/mothur_merge_count/1.39.5.0                                                                     |      86
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_otus/mothur_get_otus/1.39.5.0                                                                           |      85
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_phylo_diversity/mothur_phylo_diversity/1.39.5.0                                                             |      84
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_normalize_shared/mothur_normalize_shared/1.39.5.0                                                           |      84
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_parse_list/mothur_parse_list/1.39.5.0                                                                       |      83
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_list_otulabels/mothur_list_otulabels/1.39.5.0                                                               |      81
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_otus/mothur_remove_otus/1.39.5.0                                                                     |      81
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_consensus_seqs/mothur_consensus_seqs/1.39.5.0                                                               |      80
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_bin_seqs/mothur_bin_seqs/1.39.5.0                                                                           |      80
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_otulabels/mothur_get_otulabels/1.39.5.0                                                                 |      79
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_design/mothur_make_design/1.39.5.0                                                                     |      78
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_sens_spec/mothur_sens_spec/1.39.5.0                                                                         |      77
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_otulist/mothur_get_otulist/1.39.5.0                                                                     |      76
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_classify_rf/mothur_classify_rf/1.36.1.0                                                                     |      75
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_cluster_classic/mothur_cluster_classic/1.39.5.0                                                             |      74
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_rabund/mothur_get_rabund/1.39.5.0                                                                       |      74
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_ccode/mothur_chimera_ccode/1.39.5.0                                                                 |      73
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_trim_flows/mothur_trim_flows/1.39.5.0                                                                       |      72
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_hcluster/mothur_hcluster/1.36.1.0                                                                           |      71
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_check/mothur_chimera_check/1.39.5.0                                                                 |      71
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_communitytype/mothur_get_communitytype/1.39.5.0                                                         |      70
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_lookup/mothur_make_lookup/1.39.5.0                                                                     |      70
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_sffinfo/mothur_sffinfo/1.39.5.0                                                                             |      69
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_otu_association/mothur_otu_association/1.39.5.0                                                             |      67
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_collect_single/mothur_collect_single/1.39.5.0                                                               |      65
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_taxsummary/mothur_merge_taxsummary/1.39.5.0                                                           |      65
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_pintail/mothur_chimera_pintail/1.39.5.0                                                             |      64
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_anosim/mothur_anosim/1.39.5.0                                                                               |      63
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_corr_axes/mothur_corr_axes/1.36.1.0                                                                         |      61
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_nmds/mothur_nmds/1.36.1.0                                                                                   |      60
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_align_check/mothur_align_check/1.39.5.0                                                                     |      60
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_libshuff/mothur_libshuff/1.39.5.0                                                                           |      58
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_sabund/mothur_get_sabund/1.39.5.0                                                                       |      57
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_dist_shared/mothur_dist_shared/1.36.1.0                                                                     |      56
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_otulabels/mothur_remove_otulabels/1.39.5.0                                                           |      56
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_cooccurrence/mothur_cooccurrence/1.39.5.0                                                                   |      55
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_mimarkspackage/mothur_get_mimarkspackage/1.39.5.0                                                       |      53
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_classify_tree/mothur_classify_tree/1.39.5.0                                                                 |      52
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_trim_seqs/mothur_trim_seqs/1.36.1.0                                                                         |      49
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_collect_shared/mothur_collect_shared/1.39.5.0                                                               |      46
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_perseus/mothur_chimera_perseus/1.39.5.0                                                             |      43
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_sfffiles/mothur_merge_sfffiles/1.39.5.0                                                               |      42
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_otu_hierarchy/mothur_otu_hierarchy/1.39.5.0                                                                 |      41
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_shhh_seqs/mothur_shhh_seqs/1.39.5.0                                                                         |      41
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_cluster_fragments/mothur_cluster_fragments/1.39.5.0                                                         |      41
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_dists/mothur_get_dists/1.39.5.0                                                                         |      41
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_bellerophon/mothur_chimera_bellerophon/1.39.5.0                                                     |      40
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_primer_design/mothur_primer_design/1.39.5.0                                                                 |      35
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_amova/mothur_amova/1.36.1.0                                                                                 |      34
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_deunique_tree/mothur_deunique_tree/1.39.5.0                                                                 |      34
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_mantel/mothur_mantel/1.39.5.0                                                                               |      34
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_shhh_flows/mothur_shhh_flows/1.39.5.0                                                                       |      33
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_group/mothur_get_group/1.39.5.0                                                                         |      31
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_homova/mothur_homova/1.36.1.0                                                                               |      31
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_dists/mothur_remove_dists/1.39.5.0                                                                   |      31
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_parsimony/mothur_parsimony/1.36.1.0                                                                         |      30
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_pcoa/mothur_pcoa/1.36.1.0                                                                                   |      30
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_cluster/mothur_cluster/1.36.1.0                                                                             |      29
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_mimarks_attributes/mothur_mimarks_attributes/1.39.5.0                                                       |      26
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_sra/mothur_make_sra/1.39.5.0                                                                           |      25
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_sub_sample/mothur_sub_sample/1.36.1.0                                                                       |      24
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_oturep/mothur_get_oturep/1.36.1.0                                                                       |      22
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_count_groups/mothur_count_groups/1.36.1.0                                                                   |      20
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_dist_seqs/mothur_dist_seqs/1.36.1.0                                                                         |      16
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_pairwise_seqs/mothur_pairwise_seqs/1.36.1.0                                                                 |      14
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_split_abund/mothur_split_abund/1.36.1.0                                                                     |      12
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_groups/mothur_get_groups/1.36.1.0                                                                       |      11
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_uchime/mothur_chimera_uchime/1.36.1.0                                                               |      11
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_seq_error/mothur_seq_error/1.36.1.0                                                                         |      10
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_lefse/mothur_lefse/1.36.1.0                                                                                 |       8
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_fastq_info/mothur_fastq_info/1.36.1.0                                                                       |       7
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_metastats/mothur_metastats/1.36.1.0                                                                         |       6
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_remove_groups/mothur_remove_groups/1.36.1.0                                                                 |       6
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_get_communitytype/mothur_get_communitytype/1.36.1.0                                                         |       5
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_phylotype/mothur_phylotype/1.36.1.0                                                                         |       3
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_summary_shared/mothur_summary_shared/1.36.1.0                                                               |       2
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_reverse_seqs/mothur_reverse_seqs/1.36.1.0                                                                   |       2
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_cluster_classic/mothur_cluster_classic/1.36.1.0                                                             |       2
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_chimera_vsearch/mothur_chimera_vsearch/1.39.5.2                                                             |       2
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_make_lefse/mothur_make_lefse/1.36.1.0                                                                       |       1
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_groups/mothur_merge_groups/1.36.1.0                                                                   |       1
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_split_groups/mothur_split_groups/1.36.1.0                                                                   |       1
 toolshed.g2.bx.psu.edu/repos/iuc/mothur_anosim/mothur_anosim/1.36.1.0                                                                               |       1