acontrol filtering of parsets: Additional criteria

[hsuyeep]

acontrol currently monitors the /opt/lofar/var/run directory on mcu001 for newly scheduled observations. Note that these files are strictly speaking not the parsets that are used to control the instrument. The actual parsets are available with the same id on ccu001.

The array config and starttime are parsed from each pseudo-parset on mcu001 in order to determine if the observation is suitable for AARTFAAC startup. During my talks with the RO, it was found that additional conditions might need to be checked:

• parsets are also created for pipeline runs (offline processing of recorded data). A keyword Observation.tasktype=[Pipeline|Observation] in the parset (and Observation.processType=Observation in the 'pseudoparsets' in the MCU machine)distinguishes between the two kinds of runs. We should ignore parsets with tasktype pipeline.
• Simultaneous observations with different parsets can be scheduled. However, these currently have a mutually exclusive subset of stations in them, available via the Observation.VirtualInstrument.stationList parameter. We should interrogate this to find out if the AARTFAAC stations are among the selected stations.
• If it seems like no observation is currently scheduled due to the absence of a parset, the stations are probably in maintenance mode. This happens quite infrequently (~1/month). If no cycle observations are scheduled, TBB observations are scheduled. These appear with their own parset which we can trigger on.
• If we need to figure out if the stations are on during a timeframe not covered by a parset, we can execute the rspct --rcu command on each station, and check if the RCU is on, and the mode it is in (this tells the array configuration).
• HBA observations can appear in a variety of settings (from http://www.astron.nl/radio-observatory/astronomers/users/technical-information/lofar-stations/lofar-stations-description-). Note that currently all HBA observations will have a singe analog beam pointing direction, having each ear point to a different direction with its analog beam is not a supported mode currently.
  • HBA_[ZERO,ONE]: Use only 24 tiles from one ear of the core stations. The other 24 are turned off. We need to make sure to flag the tiles from the relevant ear.
  • HBA_DUAL: Use both ears, but as separate 'substations' with the same analog pointing. Our impact is minimal as we tap the data before the beamforming. So, we just use all tiles.
  • HBA_JOINED: Use both ears, but beamformed as a single station. This is identical to HBA_DUAL in our perspective.
  • HBA_{*}_INNER: This is a qualifier applicable to all the above modes, but makes sense only on international stations. Basically says we need to use the inner 48 tiles of the international stations. Not applicable to us.

[Error323]

During a discussion with Ralph and Andre: for the next cycle they will generate parsets for aartfaac when the system is idle.

[hsuyeep]

Update: On practical usage, currently the system is triggering on both pipeline processes and normal observations. I had an email conversation with Richard Fallows from Science Support. The conclusions are:

• Instead of checking mcu001:/opt/lofar/var/run/MCU* files, which can contain both processing and observation parsets, and then distinguishing between them, we can monitor the main observation parsets. The primary source for these is the machine ccu001, but a consistent copy is available in mcu001:/opt/lofar/var/run/Observation.
• Parallel observations (different station configurations in the same time slots) will be carried out in the future. Each parallel configuration will have an individual parset, containing the subset of stations that they use. In this case, apart from the time and AntennaSet, we should also check the list of stations involved, by parsing the keyword 'ObsSW.Observation.ObservationControl.StationControl._hostname.
• Currently, parallel observations will use a mutually exclusive set of stations. In the future, a single station might carry out combined HBA/LBA observations (e.g. trading bandwidth/number of antennas to accommodate a subset of HBA and LBA antennas). This is still experimental, but expected to become mainstream in the future. In this case, we might be better off parsing the keywords ObsSW.Observation.Dataslots.CS00?HBA0.DataslotList or ObsSW.Observation.Dataslots.CS00?HBA0.RSPBoardList for the relevant stations and 'LBA'.

Whatever it is, I'll open up another issue to atleast reject the processing jobs, else we keep kicking the system for no reason.