Method to accelerate the FEL and aBSREL

[hongzhonglu]

Dear Sergei @spond, Recently I try to conduct FEL and aBSREL calculation in batch on the cluster. I find the speed in the calculation is quite slow. I am wondering are there strategies which could accelerate the calculation on the cluster for the two methods, which can be suitable for the Hyphy version installed by conda? Look forward to your comments. Thanks very much!

Best, Hongzhong

[spond]

Dear @hongzhonglu,

Are you specifying that these jobs make use of MPI and using HYPHYMPI executables? FEL is trivially parallelizable, so you will see dramatic speedups with those. Parts of aBSREL also benefit from MPI. Here's an example script for torque for example,

#!/bin/bash
#PBS -lnodes=4:ppn=8

#request 4 nodes with 8 processors each (32 total)

export PATH=/usr/local/bin:$PATH
source /etc/profile.d/modules.sh

# load instance specific modules (depends on the system configuration)
module load aocc/1.3.0
module load openmpi/gnu/3.1

OUTFILE=${1}.FEL.json

if [ -s $OUTFILE ]
then
echo "$OUTFILE exists"
else
mpirun -np 32 /home/sergei/hyphy-dev/HYPHYMPI LIBPATH=/home/sergei/hyphy-dev/res /home/sergei/hyphy-dev/res/TemplateBatchFiles/SelectionAnalyses/FEL.bf --alignment $1 --branches All --output $OUTFILE
fi

Best, Sergei

[hongzhonglu]

Dear Sergei @spond, Thanks for your help! I am not sure about the use of MPI as I just installed the hyphy package using conda. It seems that the direct use the followed code is very slow:

!/bin/bash

SBATCH -A C3SE2020-1-8

SBATCH -N 1

SBATCH -n 20

SBATCH -o out.txt

SBATCH -t 2-00:00:00

SBATCH --mail-user=luho@chalmers.se

SBATCH --mail-type=end

hyphy fel --alignment /c3se/NOBACKUP/users/luho/cds_align_macse_remove_stop_code/OG10566_code.fasta --tree /c3se/NOBACKUP/users/luho/unroot_tree/OG10566_aa_unroot.tre --srv Yes --pvalue 0.1 --output /c3se/NOBACKUP/users/luho/fel_result/OG10566.FEL.json So I want to find a more efficient way to accelerate the calculation. Should I use HYPHYMPI by install hyphy from source code, not using conda installation as showed in http://hyphy.org/installation/?

[spond]

Dear @hongzhonglu,

The Conda package includes HYPHYMPI. Your script uses hyphy, which does not use MPI, so execution will be slow, especially for larger alignments with more sites, since FEL analyzes each site pattern separately (instead of FEL, btw, you should almost always use MEME; more sensitivity). You need to modify your execution script to

1. Use some version of mpirun -np N HYPHYMPI arguments, where N is the number of processors to use. Set to something like 32 or 64 to start with.
2. Modify the script to request MPI resources. How do to this will depend on your cluster and job manager, but you can easily find great examples online for most commonly used environments.

Best, Sergei

[hongzhonglu]

Dear Sergei @spond, Great thanks for your suggestion. I will try it based on your suggestion. It will be nice if you could add this description on the hyphy installation by conda as I did not find the related description (it is just a small suggestion).

Best, Hongzhong

[spond]

Dear @hongzhonglu,

An excellent point. I'll add some text to the installation page on the website to clarify this. I'll keep this issue open until then.

Best, Sergei

[spond]

Dear @hongzhonglu,

I've updated the documentation as per your excellent suggestion.

Best, Sergei

[hongzhonglu]

Dear Sergei @spond, Great thanks for your wonderful work! I just try your suggested method. It could run now! Best, Hongzhong