cBio / cbio-cluster

MSKCC cBio cluster documentation
12 stars 2 forks source link

Submit filter deployed to reject jobs that exceed shutdown time on command line #381

Closed tatarsky closed 8 years ago

tatarsky commented 8 years ago

Per #370 I am trying to at least automate a bit the fact that jobs that are scheduled on the soon to be shutdown and then replaced Moab server will need to be rescheduled after that is complete.

The migration of spooled entries was non-trivial given some naming matters and so if you attempt to schedule a job that exceeds the remaining time before shutdown (currently 2 days and 23 hours or so) you will be warned and your job rejected.

It will also attempt to provide a rough "live" number of seconds remaining that you could submit with if you still want to get the cycles in.

I've only been able to figure out command line qsub parsing. All the reading of #PBS directives was getting a bit elaborate and so it will not warn you of those directives walltimes. Maybe next migration.

I will continue to email people if I see jobs in this state.

Remember you can see the remaining time before the shutdown in the command:

showres system.228

The "Start" column is the time remaining:

ReservationID       Type S       Start         End    Duration    N/P    StartTime

system.228          User -  2:23:26:40  6:11:26:40  3:12:00:00   37/1328 Sun Mar  6 08:00:00

If I have a bug in my submit script I will either fix it or roll it back out.