Closed EricR86 closed 10 years ago
Original comment by Jay Hesselberth (Bitbucket: jayhesselberth, GitHub: jayhesselberth).
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
This used to be the case. It seems Platform/IBM have changed it in recent versions.
What version of LSF are you using? What's the output of lsinfo
?
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
Original comment by Jay Hesselberth (Bitbucket: jayhesselberth, GitHub: jayhesselberth).
Here is my LSF info:
$ lsinfo -V
Platform LSF 7.0.6.134609, Sep 04 2009
Copyright 1992-2009 Platform Computing Corporation
binary type: linux2.6-glibc2.3-x86_64
I guess it would be possible to set these values based on LSF versions. I can fix and submit a pull request if you send your version info. Would be nice to know in which version of LSF this behavior changed, if that's the case.
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
Can you send it without -V
so I can see what the resource information looks like? I am wary of distinguishing based on version rather than something functional.
I no longer have access to an LSF system so I can't test this myself anymore.
Original comment by Jay Hesselberth (Bitbucket: jayhesselberth, GitHub: jayhesselberth).
Full input of lsinfo
:
#!bash
jhessel@amc-tesla ~
$ lsinfo
RESOURCE_NAME TYPE ORDER DESCRIPTION
r15s Numeric Inc 15-second CPU run queue length
r1m Numeric Inc 1-minute CPU run queue length (alias: cpu)
r15m Numeric Inc 15-minute CPU run queue length
ut Numeric Inc 1-minute CPU utilization (0.0 to 1.0)
pg Numeric Inc Paging rate (pages/second)
io Numeric Inc Disk IO rate (Kbytes/second)
ls Numeric Inc Number of login sessions (alias: login)
it Numeric Dec Idle time (minutes) (alias: idle)
tmp Numeric Dec Disk space in /tmp (Mbytes)
swp Numeric Dec Available swap space (Mbytes) (alias: swap)
mem Numeric Dec Available memory (Mbytes)
root Numeric Dec HPC ELIM
maxroot Numeric Dec HPC ELIM
processes Numeric Dec HPC ELIM
clockskew Numeric Dec HPC ELIM
ncpus Numeric Dec Number of CPUs
ndisks Numeric Dec Number of local disks
maxmem Numeric Dec Maximum memory (Mbytes)
maxswp Numeric Dec Maximum swap space (Mbytes)
maxtmp Numeric Dec Maximum /tmp space (Mbytes)
cpuf Numeric Dec CPU factor
rexpri Numeric N/A Remote execution priority
nprocs Numeric Dec Number of physical processors
ncores Numeric Dec Number of cores per physical processor
nthreads Numeric Dec Number of threads per processor core
server Boolean N/A LSF server host
LSF_Base Boolean N/A Base product
lsf_base Boolean N/A Base product
LSF_Manager Boolean N/A LSF Manager product
lsf_manager Boolean N/A LSF Manager product
LSF_JobSchedu Boolean N/A JobScheduler product
lsf_js Boolean N/A JobScheduler product
LSF_Make Boolean N/A Make product
lsf_make Boolean N/A Make product
LSF_Parallel Boolean N/A Parallel product
lsf_parallel Boolean N/A Parallel product
LSF_Analyzer Boolean N/A Analyzer product
lsf_analyzer Boolean N/A Analyzer product
mips Boolean N/A MIPS architecture
sparc Boolean N/A SUN SPARC
hpux Boolean N/A HP-UX UNIX
aix Boolean N/A AIX UNIX
irix Boolean N/A IRIX UNIX
rms Boolean N/A RMS
pset Boolean N/A PSET
dist Boolean N/A DIST
slurm Boolean N/A SLURM
cpuset Boolean N/A CPUSET
solaris Boolean N/A SUN SOLARIS
fs Boolean N/A File server
cs Boolean N/A Compute server
frame Boolean N/A Hosts with FrameMaker licence
bigmem Boolean N/A Hosts with very big memory
diskless Boolean N/A Diskless hosts
alpha Boolean N/A DEC alpha
linux Boolean N/A LINUX UNIX
nt Boolean N/A Windows NT
mpich_gm Boolean N/A MPICH GM MPI
lammpi Boolean N/A LAM MPI
mpichp4 Boolean N/A MPICH P4 MPI
mvapich Boolean N/A Infiniband MPI
sca_mpimon Boolean N/A SCALI MPI
ibmmpi Boolean N/A IBM POE MPI
hpmpi Boolean N/A HP MPI
sgimpi Boolean N/A SGI MPI
intelmpi Boolean N/A Intel MPI
crayxt3 Boolean N/A Cray XT3 MPI
crayx1 Boolean N/A Cray X1 MPI
mpich_mx Boolean N/A MPICH MX MPI
mpichsharemem Boolean N/A MPICH Shared Memory
mpich2 Boolean N/A MPICH2
mg Boolean N/A Management hosts
openmpi Boolean N/A OPENMPI
bluegene Boolean N/A BLUEGENE
define_ncpus_ Boolean N/A ncpus := procs
define_ncpus_ Boolean N/A ncpus := cores
define_ncpus_ Boolean N/A ncpus := threads
Platform_HPC Boolean N/A platform hpc license
platform_hpc Boolean N/A platform hpc license
fluent Boolean N/A fluent availability
ls_dyna Boolean N/A ls_dyna availability
nastran Boolean N/A nastran availability
pvm Boolean N/A pvm availability
openmp Boolean N/A openmp availability
ansys Boolean N/A ansys availability
blast Boolean N/A blast availability
gaussian Boolean N/A gaussian availability
lion Boolean N/A lion availability
scitegic Boolean N/A scitegic availability
schroedinger Boolean N/A schroedinger availability
hmmer Boolean N/A hmmer availability
type String N/A Host type
model String N/A Host model
status String N/A Host status
hname String N/A Host name
TYPE_NAME
UNKNOWN_AUTO_DETECT
DEFAULT
DEFAULT
CRAYX1
DigitalUNIX
ALPHA5
ALPHASC
HPPA
IBMAIX532
IBMAIX564
LINUX
LINUX2
LINUXAXP
LINUX86
LINUXPPC
LINUX64
DLINUX
DLINUX64
DLINUXAXP
SLINUX
SLINUX64
NECSX6
NECSX8
NTX86
NTX64
NTIA64
SGI6
SUNSOL
SOL732
SOL64
SGI64
SGI65
SGI658
SOLX86
SOLX8664
HPPA11
HPUXIA64
MACOSX
LINUXPPC64
LINUX_ARM
X86_64
SX86_64
IA64
DIA64
SIA64
MODEL_NAME CPU_FACTOR ARCHITECTURE
Intel_EM64T 60.00 x15_6789_IntelRXeon
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
This says mem
is Available memory (Mbytes)
. But on your system it is actually Gbytes? So I can't get the information by parsing lsinfo
.
The change in documentation happened around LSF 9. The Platform LSF Configuration Reference section on lsf.conf includes this for LSF_UNIT_FOR_LIMITS:
This parameter alters the meaning of all numeric values in lsb.resources to match the unit set, whether gpool, limits, hostexport, etc. It also controls the resource rusage attached to the job and the memory amount that defines the size of a package in GSLA.
There's nothing about resources in the LSF 7.0.6 version of that document.
Similarly, Administering Platform LSF 9.1.1's section on Load indices includes several new mentions of LSF_UNIT_FOR_LIMITS
within guillemets, and change lines within the PDF version, which I presume is IBM's way of saying this section has changed.
I don't know whether the change was due to different behavior in LSF 9, or what is effectively a doc bug in LSF 7. How do you know that select[mem>2148 && tmp>48]
asks for >2148 GB on your system?
Original comment by Jay Hesselberth (Bitbucket: jayhesselberth, GitHub: jayhesselberth).
On our system the select statement uses the LSF_UNIT_FOR_LIMITS, which in our case is GB; I guess this is what the doc you cite refers to.
I guess I don't know per se what it's requesting, but for everything else I use GB size requests (i.e. select[mem>8] identifies nodes with 8 GB available), and segway jobs won't run with the setting at 2148; changing to GB in cluster.lsf.py lets them run.
-- Jay Hesselberth Assistant Professor Biochemistry & Molecular Genetics University of Colorado School of Medicine off: (303) 724-5384 lab: (303) 724-5486 jay.hesselberth@gmail.com
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
Can you get me the result of bjobs -l
for a stalled job due to requesting select[mem>2148]
?
Original comment by Jay Hesselberth (Bitbucket: jayhesselberth, GitHub: jayhesselberth).
Here's a test script that stays in PEND state.
#!bash
jhessel@amc-tesla ~/devel/segway
$ bjobs -l
Job <510660>, Job Name <test>, User <jhessel>, Project <default>, Status <PEND>
, Queue <normal>, Job Priority <50>, Command <#! /usr/bin/
env bash; #BSUB -J test;#BSUB -R "select[mem>2148] rusage[
mem=2148]"; sleep 100>
Fri Mar 28 14:36:05: Submitted from host <amc-tesla>, CWD <$HOME/devel/segway>,
Requested Resources <select[mem>2148] rusage[mem=2148]>;
PENDING REASONS:
Not specified in job submission: 3 hosts;
Job's resource requirements not satisfied: 14 hosts;
Closed by LSF administrator: 1 host;
SCHEDULING PARAMETERS:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - 1.5 1.5 - - - - - - - -
loadStop - 3.0 3.0 - - - - - - - -
root maxroot processes clockskew
loadSched - - - -
loadStop - - - -
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
The reference for bsub
in LSF 7.0 (2007) says it should honor LSF_UNIT_FOR_LIMITS
for -R
. LSF 6.0 doesn't seem to have LSF_UNIT_FOR_LIMITS
. I am going to assume I got this one wrong.
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
I've checked in a fix to the issue2 branch, ac94de1674933292780ff5fd5251626a3bbc8328. If this fixes it for you, I will merge to default.
Original comment by Jay Hesselberth (Bitbucket: jayhesselberth, GitHub: jayhesselberth).
Fixed, LSF jobs run.
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
close issue2. issue #2
→ \<\<cset d2932c2127a9ef8141ddd9a463e4079ccc6325fa>>
Original report (BitBucket issue) by Jay Hesselberth (Bitbucket: jayhesselberth, GitHub: jayhesselberth).
Our LSF system is set up with GB as the LSF_UNIT_FOR_LIMITS:
And segway.cluster.lsf parses this correctly as the DIVISOR_FOR_LIMITS from our lsf.conf file.
However, for the resource request, the DIVISOR_FOR_LIMITS is ignored and instead is specified in MB, irrespective of the configured unit:
So on our system, segway is asking for 2148 GB, which is awesome, but exceeds that RAM on our compute blades. The jobs stay in PEND status and never run.
This comment in lsf.py::make_res_req() suggests that resources are always specified in MB:
I looked at that document, and don't see the logic for always specifying resources in MB.
So:
Updating to the following in segway.cluster.lsf lets my jobs run:
Does the comment suggest that we are somehow inappropriately managing LSF? I can't change LSF_UNIT_FOR_LIMITS, it would affect many users.