Installing dependancies on compute nodes during "module load" can cause SGE job to exceed memory limit

alces-software / flight-aws-marketplace

Templates for flight in AWS marketplace

GNU Affero General Public License v3.0

2 stars 0 forks source link

Installing dependancies on compute nodes during "module load" can cause SGE job to exceed memory limit #76

Closed wdm78 closed 8 years ago

wdm78 commented 8 years ago

To replicate:

Launch a new cluster
Use qconf -sc to show default memory limit (usually 1.5GB for c4 instances)
Install "main/apps/R"
Write a simple jobscript like this:

#!/bin/bash -l
#$ -t 1-50
module load apps/R
R --version

Submit the script
Check /var/spool/gridscheduler/execd/node*/messages file to see jobs that exceeded memory limit and were killed

mjtko commented 8 years ago

I think we diagnosed this as being due to yum consuming vast amounts of RAM while generating package indices. Perhaps this can be worked round by causing yum to generate package indices at boot?

mjtko commented 8 years ago

Addressed in https://github.com/alces-software/clusterware-handlers/commit/3552ceaeb5765bf93d8c9595fb09f6affcc61051 and https://github.com/alces-software/clusterware-handlers/commit/b09c5c1fe2921bac0c9ae5816cb2cdff9682f204

vlj91 commented 8 years ago

This is included in 2016.3