mantis interface request interval determination

[laroque]

I saw that in slack there was some discussion about what the "right" way to determine the interval for sending requests to mantis are. I'll summarize how it currently works and why, then welcome comments.

First there is a base DAQProvider class which implements three important methods:

• start_run(run_name) takes care of getting a run_id from the database interface
• start_timed_run(run_name, run_time) schedules a call to end_run() after run_time seconds, then calls start_run()
• end_run() resets internal parameters to the default/stopped state and cancels any scheduled end_run() timeouts (this allows you to use end_run on one started with start_run or start_timed_run without issue)

Next, each there is a class which inherits from DAQProvider for each DAQ system. They override the parent start_run() and end_run() but are expected to use super() to still call the parent methods. This means each interface implements whatever actually needs to happen but the above behaviors are done the same across all DAQ systems.

In the case of the interface for mantis, the class also inherits from core.Spime (which allows it to behave like a sensor logger). The "logging" action for mantis is configured to be sending an OP_RUN to the mantis queue with appropriate parameters. This results in in an OP_RUN being sent every log_interval seconds. Aside from some unfortunate names (like log_interval) this is nice because it uses the same code and pika based asynchronous callback system to take care of timing those events (which means there aren't any blocking calls to things with long while loops or ugly threading blocks to deal with).

Other considerations:

• I want data taken with start_run() followed by end_run() some amount of time later, to be indistinguishable from data taken with start_timed_run(), assuming the right run_time is used, in terms of data quality. This means, for mantis, basing the log_interval on the total run_time is awkward.
• One could consider initially sending two requests to mantis, that way there is always one sitting in the queue when an acquisition finishes and possibly deadtime is reduced. Starting a timed run could account for this extra time in a way that endrun wouldn't (the latter would tend to produce one more acquisition than a timed run), but at least the timing between acquisitions would still be consistent. (Note to self: If we want to do this is should be as trivial as adding the line self.on_get() into MantisAcquisitionInterface.start_run after the call to super but before setting loggingstatus to 'on')

[nsoblath]

I think we should consider the other side of the issue, what would be ideal considering the way that Mantis and the digitizer function, and then we can determine the best way to implement it.

Mantis's mode of operation is to run for a fixed amount of time to create a single data file (originally this was the sense of a "run," though now a run can consist of multiple of such files). We can consider how we would want to perform both timed and untimed runs:

• Timed We want to take more-or-less continuous data for X seconds. This will be mainly made up of Mantis acquisitions each lasting Y seconds, where Y <= X. The last acquisition in the run might be shorter than Y, if X % Y != 0.
• Untimed We want to take more-or-less continuous data for some unknown amount of time, until the run is explicitly ended. Acquisitions lasting Y seconds should be taken until the run is canceled.

The DAQ interface can set the acquisition time in mantis, so we don't have to worry about the mantis settings not being synchronized with the interface settings. Furthermore, that allows us to change the acquisition time between acquisitions if that's appropriate.

Here are my thoughts on how I would implement the two modes in the DAQ interface. These are what I think I would ideally want, and therefore they might not take into account some of the considerations you mentioned above:

• Timed start_timed_run() would determine the set of acquisition times needed and submit them all at once. A record would be kept of the runs submitted. end_run() would then cancel all of the runs; any that actually remain in the queue would be canceled (this could be made a bit smarter by querying the server about the status of the runs, starting with the last and going towards the first, canceling each until it finds one not still in the queue).
• Untimed I would have run commands sent every Y seconds, but ideally I would also make the interval between the first run request and the next slightly shorter than Y so that all future run requests are sent slightly before they're needed.

[laroque]

Okay, so some follow up thoughts:

For the untimed case. Do you mean that the time between requests being sent L == Y - epsilon; or that L == Y, starting with the time between runs 1 and 2, but that between run 0 and run 1 the time is somewhat less than L? The former case seems like it could really build up over something like an overnight run; the latter case is basically what I had suggested above (take to the extreme where that first time delta is 0).

I guess my concerns was something more like: "If in both timed and untimed cases, what we really want is more-or-less continuous data, is it reasonable that one case works in a way that could be more continuous than the other?" If it ever matters then this seems like a potential gotcha that we will screw up in the future.

For the end_run() cancel stuff. What is the current mantis_server API.

• Are queued acquisitions referenced by the output file name or something else?
• If something else, does and/or could it respond to my request message with that reference in the payload?
• Does there exist either a dump_all_queued() or drop_acquisitions([list, of, files, to, drop]) or is it only drop_acquisition()?
• Did sending broadcasts when acquisitions complete ever get implemented or was it dropped when fsnotify was put into hornet? In the untimed case, if it is going to keep a list of acquisitions it has requested (so it can kill them if need be), it would be nice to drop them as they finish so that that list doesn't grow without bound... Or were you imagining not keeping such a list, and instead asking mantis_server for everything in the queue in the event of an end_run and then killing whatever is there? It isn't obvious when this would ever be an actual problem, but that behavior would potentially cancels acquisitions requested by mantis_client or some other process.

[nsoblath]

For the untimed case, I definitely meant that starting between runs 1 and 2 L==Y, but that between 0 and 1, L < Y. I don't know if that fits with the setup that's currently used, where the DAQ interface for mantis behaves like a logger.

Acquisitions are referenced by a UUID that gets assigned when the acquisition request is received, and is sent in the reply to that request.

Currently the Mantis API includes the ability to dump all acquisition requests, or to cancel individual requests (keyed by the UUID).

The broadcast about completed files is there. One just needs to specify the completed-file-key in the server config. This is left un-set by default, which turns it off. I hadn't actually thought of that as a way to limit the list of acquisitions that would need to be canceled, but that would work perfectly.

[laroque]

I've implemented the untimed-case which I think is consistent with both of our suggestions (based on my note-to-self in the original issue description). The model will be that when you start_run() an acquisition request is sent immediately, then "logging_status" is set to on, which sends an acquisition request immediately, and then another ever log_interval seconds.

The result is that acquisition request 1 is send epsilon seconds after acquisition request 0. For all i>1, acquisition request i is sent log_interval seconds after acquisition request i-1.

Remaining items are:

• [x] Have end_run() call the mantis method to dump all acquisition requests by sending an OP_CMD to .stop-all (this stops any running acquisition and clears the queue). The payload format here needs to be clarified.
• [x] Have timed run go ahead and send the full set of requests for the time requested. This will be easy enough to put in, but I don't want to do it until after end_run() is done.

[nsoblath]

Based on our discussion today on Slack, all of this sounds good.

For the record, I raised a question about whether we should be using stop-all when mantis can receive input from multiple sources and regarding different run configurations. We agreed that this amount of freedom is bad, considering that there is only a single run queue, and requests are all handled serially, regardless of the source. There is now a feature request for Mantis regarding a lockout feature, but for now we'll rely on the small number of people using Mantis maintaining good practices, and only having one user of Mantis at a time.

[laroque]

merged into develop from feature/mantis_interface_update branch in 7646b0c039285