LLNL / serac

Serac is a high order nonlinear thermomechanical simulation code
BSD 3-Clause "New" or "Revised" License
178 stars 31 forks source link

Give jobs a deadline on Quartz #1118

Closed chapman39 closed 3 months ago

chapman39 commented 3 months ago

With this PR, CI jobs now have a deadline on Quartz. This way, if jobs get stuck on the queue after gitlab fails, it won't clog up the system. I choose an hour because that's the same deadline as Gitlab.

If --deadline=now+1hour and -t25, the job will dequeue after 35 minutes (60-25).

I couldn't find a similar argument on lalloc/ bsub commands (Lassen).

Also, I snuck in some more copyright updates.

chapman39 commented 3 months ago

We can change the deadline time to whatever makes sense. We could also use the immediate option instead.

Sort of a funny way to test this argument is by allocating 60 minutes and setting the deadline to an hour from now - requiring you immediately get an allocation.

[meemee@ruby965:repo]$ s1 -t60 --deadline=now+1hour echo "hello"
srun -N1 -Aeng -t60 --deadline=now+1hour echo hello
srun: job 2630692 queued and waiting for resources
srun: Job has been cancelled
srun: error: Unable to allocate resources: Job/step already completing or completed
codecov-commenter commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 90.63%. Comparing base (dad99cd) to head (cf8ab9d).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop #1118 +/- ## ======================================== Coverage 90.63% 90.63% ======================================== Files 162 162 Lines 14464 14464 ======================================== Hits 13109 13109 Misses 1355 1355 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.