Same scheduled LOFAR observation results in multiple triggers of control system.

transientskp / aartfaac-control

AARTFAAC control scripts

0 stars 0 forks source link

Same scheduled LOFAR observation results in multiple triggers of control system. #32

Closed hsuyeep closed 9 years ago

hsuyeep commented 9 years ago

emails within a few 10s of secs are generated from acontrol showing the successful startup of processes to handle the same obsid. NOTE: I have changed the glob string from MCU* to Observation* in the check for new observations in /opt/lofar/var/run

Another, slightly related issue is the appearance of what seems like multiple short observations, resulting in multiple emails with different obsids. This could be due to multiple observations with overlapping start times. Not sure if a check for such an overlap is made in acontrol

hsuyeep commented 9 years ago

The latter were actually many, many short (~2min) observations. Not sure if this is a test phase, and will recur or not, but maybe we should have a check on the observation duration, and not start our processes for observations shorter than 15mins.

hsuyeep commented 9 years ago

On further checking, it looks like we should also check if the stations involved in a particular observation are AA-6/12 or not. Eg. Observation [OBS - LBA_INNER 57.6MHz 21:15-21:25] /opt/lofar/var/run/Observation352204 [Triggered 5 times, once a min.] Observation [OBS - LBA_INNER 57.6MHz 21:15-05:55] /opt/lofar/var/run/Observation352208 [Triggered 6 times, once a min.]

hsuyeep commented 9 years ago

Also noted that for the multiple short observations, often they are in the same array configuration. So maybe an approach is to maintain state, and if triggered again in the same configuration, just continue without stopping any processes.

hsuyeep commented 9 years ago

Fixed with commit 7f8df02b by:

checking ObsSW....StationControl._hostname for a maintained list of AARTFAAC stations. Noted that even the Observation* files can have processing descriptions which can run simultaneously with the observations, but which do not have the above keyword, and are hence filtered out.
Restricting the scheduled observations to those with a duration longer than 10 mins.

TODO: Try to figure out what observation we are currently running, if any, and if the new (short) observation has the same array configuration, simply continue. Requires maintaining state across parset file parsing.