rafwiewiora
commented
8 years ago Ok I did a trial MPI run and it's not all quite right :/
I did 2 CPU threads and 2 GPUs:shared. Log is here - you can see it appears like there are two processes running everything in parallel, rather than dividing work in half - there's even a time where one deletes a file, and the other one complains when it tries the same next. Log: https://gist.github.com/rafwiewiora/b7f663ee76059ea8aca22b42347cafce (killed it at the start of explicit refinement - hanged).
I did:
Ensembler script:
# add anaconda to PATH
export PATH=/cbio/jclab/home/rafal.wiewiora/anaconda/bin:$PATH
ensembler init
cp ../manual-overrides.yaml .
ensembler gather_targets --gather_from uniprot --query SETD8_HUMAN --uniprot_domain_regex SET
ensembler gather_templates --gather_from uniprot --query SETD8_HUMAN
# no loopmodel
ensembler align
ensembler build_models
ensembler cluster --cutoff 0
ensembler refine_implicit --gpupn 4
ensembler solvate
ensembler refine_explicit --gpupn 4MPI script:
# walltime : maximum wall clock time (hh:mm:ss)
#PBS -l walltime=03:00:00
#
# join stdout and stderr
#PBS -j oe
#
# spool output immediately
#PBS -k oe
#
# specify GPU queue
#PBS -q gpu
#
# nodes: number of nodes
# ppn: number of processes per node
# gpus: number of gpus per node
# GPUs are in 'exclusive' mode by default, but 'shared' keyword sets them to shared mode.
#PBS -l nodes=1:ppn=2:gpus=2:shared
#
# export all my environment variables to the job
#PBS -V
#
# job name (default = name of script file)
#PBS -N myjob
#
# mail settings (one or more characters)
# email is sent to local user, unless another email address is specified with PBS -M option
# n: do not send mail
# a: send mail if job is aborted
# b: send mail when job begins execution
# e: send mail when job terminates
#PBS -m n
#
# filename for standard output (default = <job_name>.o<job_id>)
# at end of job, it is in directory from which qsub was executed
# remove extra ## from the line below if you want to name your own file
##PBS -o myoutput
# Change to working directory used for job submission
cd $PBS_O_WORKDIR
# add anaconda to PATH
# export PATH=/cbio/jclab/home/rafal.wiewiora/anaconda/bin:$PATH
# Set CUDA_VISIBLE_DEVICES for this process
python build-mpirun-configfile.py bash ensembler_script.sh
# Launch MPI job.
mpirun -configfile configfileThis uses the build-mpirun-configfile.py from https://github.com/choderalab/clusterutils/blob/master/clusterutils/build_mpirun_configfile.py
Any pointers?
jchodera
I know this would take me ages to work out on my own, so hopefully someone could spare a few mins for a tutorial please.
How does one set up Ensembler to run with multiple GPUs? I have 23 models and need to equilibrate them all for 5ns each.
More detailed questions:
--gpupntorefine_explicitand everything else will happen automagically as far as Ensembler is concerned?Thanks!