search

LSSTDESC / ImageProcessingPipelines

Alert Production and Data Release image processing pipelines using the LSST Stack
BSD 3-Clause "New" or "Revised" License
3 stars 2 forks source link

SRS Pipeline issues at NERSC for mini DRP #29

Open heather999 opened 6 years ago

heather999 commented 6 years ago

At NERSC we are sharing "CORIP" to run pilot jobs. There is a job limit, default 90000. With the ongoing phoSim Run 1.1p jobs, we are already at this limit. This prevents large scale testing of ingests, etc. I need some input from Tony and Tom to see how high we can set this limit and how to prevent any interference with the higher priority phoSim jobs.

heather999 commented 6 years ago

Working this out with Tony and Tom in the background. For now, Tom has kindly modified some of the parameters associated with CORIP: JobsSubmittedLimit 500,000 JobsSubmittedLimitPerTask 40,000 JobsTotalLimit 500,000 JobsTotalLimitPerTask 40,000 This has allowed my jobs to run. When the mini-DRP pipeline gets going, and is running at the same time as phoSim production - we will likely need a separate pilot daemon for the descdm account.