search

dmwm / CRAB2

CRAB2
2 stars 11 forks source link

propagate to JDL the request for more memory #815

Closed ericvaandering closed 10 years ago

ericvaandering commented 10 years ago

Original Savannah ticket 97656 reported by belforte on Thu Sep 20 12:09:43 2012.

at least for glidein condor JDL at least for remoteGlidein see https://hypernews.cern.ch/HyperNews/CMS/get/crabDevelopment/1611/1/2/2.html

ericvaandering commented 10 years ago

Comment by belforte on Thu Nov 8 02:05:57 2012

from Igor:

Yes... we can match on memory size. And no need to re-invent the wheel... condor has already a job attribute for this: RequestMemory http://research.cs.wisc.edu/condor/manual/v7.8/11_Appendix_A.html#86575

Now, for the other half of the equation: Glideins already advertise how much memory the job is allowed to use (set by the factory); so it is easy to match on that. We just need to tweak the frotnend config a bit (we are not matching on that right now).

And Condor also pushes an attribute called ResidentSetSize in the Job ClassAd, so we can kill/hold jobs that go past the limit easily.

In summary, we can deal with most of it in the glideinWMS/Condor world. All you need to do in the CRAB domain is to add request_memory = <nr MBytes> and things will/should work from there on.

ericvaandering commented 10 years ago

Comment by belforte on Thu Nov 8 09:32:23 2012

done, tested, committed

new paramenter in crab.cfg is max_rss=<MBytes> it is only relevant for remoteGlidein since for other schedulers I did not find a clean way to not change the watchdog. If someone wants to submit via another scheduler and override the preset 2.3 GB limit they will need to use the additional input file trick.

/local/reps/CMSSW/COMP/CRAB/python/cms_cmssw.py,v <-- cms_cmssw.py new revision: 1.390; previous revision: 1.389

/local/reps/CMSSW/COMP/CRAB/python/SchedulerRemoteglidein.py,v <-- SchedulerRemoteglidein.py new revision: 1.8; previous revision: 1.7

ericvaandering commented 10 years ago

Closed by belforte on Thu Dec 20 08:01:49 2012

ericvaandering commented 10 years ago

Comment by belforte on Thu Dec 20 08:01:49 2012

released in CRAB 2_8_4