search

dessn / Pippin

Pipeline for photometric SN analysis
MIT License
9 stars 10 forks source link

time-stamp check to prevent multiple Pippin launches from interfering #78

Open RickKessler opened 2 years ago

RickKessler commented 2 years ago

Before re-launching pippin, one must remember to kill previous pippin job that may still be running in the background. In case we forget to kill previous pippin job, here is a suggestion for how to do this automatically. First, each pippin launch should write a launch time-stamp to a file in the PIPPIN_OUTPUT directory. Next, every time pippin wakes up to check status, it should verify that the time-stamp in the file hasn't changed. If there was another relaunch, the original pippin job will find a mis-matched time stamp and should quit ... perhaps leaving a PIPPIN.STOP file to inform the user.

OmegaLambda1998 commented 2 years ago

I believe there's a better way of doing this.

  1. If a task fails, it should cancel it's job regardless. This can be added as a continuation of 90db9e9 by defining a method for each task which will kill that task's batch jobs
  2. This will help for cancelling a pippin job via Ctrl-c, however if the job dies unexpectedly, such as a terminal closing or ssh connection dying without Pippin being safely in a screen, this won't help. As such it would be good to have pippin place itself in a screen automatically, and close that screen once it's done.