Emulators: provide a backdoor

[IsabellaRey]

As a developer looking to write IOC unit tests, I would like emulators to listen to "backdoor" commands so we can change its state/settings at runtime.

Take the emulator for cryo valve created as part of ISISComputingGroup/IBEX#1524 and make sure it can be controlled via a telnet "backdoor". Ideally the backdoor port should be different from the one where the IOC connects. (Make sure the two ports are talking to the same "instance" of the emulator: if you change a value from one port, you should be able to see the effect of the change on the other port)

[OwenArnold]

This is something that we are currently designing for as part of an ongoing design issue https://github.com/DMSC-Instrument-Data/documents/pull/6. The intention is to allow a separate process to run to manage the simulation "Environment" without having to restart the simulation.

[OwenArnold]

@IsabellaRey How important is is that the "backdoor" is controllable via telnet over some other simple protocol. I imagine that having access to the "backdoor" to control the simulation and it's parameters is more important that the actual protocol?

We have another possibility proposed by the ESS that would allow updates in the form control.py -h host -p port parameter value which might actually be simpler than using a telnet client.

[IsabellaRey]

@OwenArnold I said telnet because that's what we have in our embrionic emulators currently, and I thought Mike's emulators were using that too. It doesn't have to be.

Whatever allows us to connect to the emulator easily via i) some manual process we may want to use for manual checks, ii) an IOC PV in case we decide to have a dedicated PV to send backdoor commands, and iii) potentially a python script, to leave options open!

[OwenArnold]

Thanks for clarifying @IsabellaRey. With Plankton, we are most likely going to cover (i) as a priority. (ii) and (iii) look to me more like nice to have features rather than essentials. Is that fair to say? Let me know otherwise.

And if we are interpreting your requirements correctly you need to control both the simulation environment (delta-t, connect, disconnect etc) as well as parameters that are bespoke to a given simulator (temperature set point etc)?

[IsabellaRey]

@OwenArnold in answer to your second question: the background story here is that we plan to use emulators to run automated IOC tests at first, so for that we can start with fairly dumb emulators whose state, params (like temperature readbacks and alarm status), and connection status can be controlled via the backdoor. More realistic (as in, device-like) emulators will be useful in a second stage, for example in validating user scripts and estimating script duration (varying delta-t somehow here would be useful, though maybe it doesn't have to be varied at runtime).

In answer to your first question: our plan would be to somehow have the IOC itself launch the emulator (when in DEVSIM mode), by using some kind of wrapper script that ideally would find a free port, launch the emulator on that port, and pass the port number back to the IOC in the form of a macro (Stories #1529 and #1530). This way the port handling happens all inside the IOC. As for the backdoor, the key point for unit tests is that we have an easy way for the tests to send backdoor commands, in an automated way. So the test could either:

• connect directly to the emulator (so we need a way to do it e.g. in python, case (iii); we could simply use python subprocess); the problem here is that we need to get the port number to the unit test (otherwise we might have to hard-code it)

or:

• send the backdoor commands via a dedicated PV in the IOC. This way the backdoor port is handled within the IOC in the same way as the main one, and we don't have to pass it outside. This is case (ii)

We thought case (ii) might be the most straight-forward, but we're open to suggestions for alternatives!

[MikeHart85]

@IsabellaRey

Controlling / backdooring into the simulation

As Owen mentioned, we're currently leaning towards adopting the approach taken by the live-data-prototype, with some modifications. You can see the live-data-prototype implementation here: https://github.com/DMSC-Instrument-Data/live-data-prototype/blob/master/control.py

For manual control, this would mean there will be a python script you can execute from the command line which can connect to a running simulation to list, get and set available variables, based on command-line arguments.

If you're writing a Python script, you could instead import the controller modules and do all the same things programmatically. So if your tests are written in Python, this would be a convenient way to access the backdoor without needing to launch a subprocess.

The only thing this wouldn't easily support is using PVs to send backdoor commands. Would this work for you?

Approach to testing

When running your unit tests, do you know whether you will be restarting the IOC between individual tests? Or will you start the IOC once and then run multiple tests against it?

If you'll be restarting the IOC between tests, that likely means the emulation will/can also be restarted. Plankton has the concept of Setups, which are selected at startup and can customize how an emulated device behaves. The idea behind this was to allow emulating a faulty or modified device.

The way we had imagined Plankton being used for tests like this is that it just sits behind the IOC and the test never needs to manipulate the emulated device directly (apart from maybe adjusting delta t speed). It would just work with the IOC, as if it was connected to a real device. If you wanted to, for example, test that a temperature PV works correctly, you would do this by instructing the IOC to heat to some target temperature and observe what happens to the temperature readout (as opposed to overriding the temperature readout through a backdoor).

What do you guys think about this kind of approach compared to using a backdoor? Is the backdoor just something you're looking to use before more detailed emulators are available?

[IsabellaRey]

@OwenArnold @MikeHart85 As this is getting lengthy and prone to misunderstanding, I'll arrange a catchup with you guys so we can all be on the same page!

[MichaelWedel]

@IsabellaRey, if you're interested, a first step for this functionality is on its way in plankton (see pull request). At the moment this exposes the simulated device directly through JSON-RPC via ZMQ (i.e. not through EPICS, TCP stream, modbus or any other device specific protocol), in addition to whatever protocol the device uses. On the client side there is a small script control.py which allows accessing the device via the command line, but it's also possible to write Python scripts which access the device object on the server-side transparently, using provided client classes.

At the moment you still have to tell simulation.py and control.py the host/port over which this is happening. Do I understand correctly that ideally you would like to have some sort of "I don't care, just use any port and let me know which one" mode?

[IsabellaRey]

Thanks, @MichaelWedel

Regarding passing port numbers: we think it would be OK if we tell the emulator on which ports to run, both for the main TPC comms to the IOC, and the backdoor. This way we’re more flexible, we can always write a wrapper script ourselves to find an available port.

Does this answer your question?

[MichaelWedel]

Yes, so for the time being specifying ports for both communication channels seems to be sufficient, so we'll probably leave it at that and check again if/when requirements change. Thanks!

[IsabellaRey]

To be reviewed with the pull request for ISISComputingGroup/IBEX#1524

[AdrianPotter]

Backdoor has been confirmed available as part of #1524 and can be operated as expected