automated data collection

[prjemian]

Provide a bluesky plan for automated data collection triggered by an EPICS PV.

Expectations are:

• the automated plan is started in the RunEngine and sits idle
• remote user creates a scan specification (such as text or spreadsheet file)
• that specification is posted, via EPICS, to the automated plan
• automated plan, monitoring that PV, acts on the plan, then returns to idle
• automated plan can be exited by ^C from console or by special command in the trigger (or other) PV

---

Here are Jan's thoughts on this for a SPEC implementation:

I talked with Pete on this for BlueSky briefly, but considering the state of BS for now and some device support we may not have ready yet, I need also spec support on this. And this is first time I am trying to write specifications on what I think should work for us… First - there may be better way, this is just my ideas, if there is existing support, let’s use that and not write anything new. Second - I think I could whip this up in spec (not BS) but I think you should be able to do quicker and BETTER job.

---

Description Recall how we collect data today - we write out command file (let’s call it usaxs.mac) using Python tool or manually and then let users issue in command line command : CollectData usaxs.mac

In order to make USAXS suitable for remote operations I want to prevent remote users from access to command line. BUT, I want staff to have standard access to current spec session, so staff can do what they want as needed… I therefore need something which will execute a command file or few spec commands when told by users via GUI, nothing else. My proposal is to do this through epics.

Add epics BO “StartAutomaticCollection” and Stringout “AutomaticCollectionStringInput” .

before data collection user would (using qtdm GUI, some script, or using a python program) write name of the command file or command itself in “ AutomaticCollectionStringInput” and after short delay (to flush buffers) would set “StartAutomaticCollection” to 1.

Now, in spec - or BS - we need function -- let’s call it “AutoCollectData” — started by staff when instrument is handed over to users and which will do following

Go in sleep loop and every 1-2 seconds check on “StartAutomaticCollection”, if “StartAutomaticCollection” = 0, stay in sleep loop. Print message in terminal so everyone knows that this function is running, may be including “use ctrl-C to stop”. if “StartAutomaticCollection” = 1 code will do following:

1. check “ AutomaticCollectionStringInput”. 2.a. if it is name of valid file in current data folder, 2.a.1 set “StartAutomaticCollection” = 0 2.a.2 issue CollectData $“ AutomaticCollectionStringInput” command (e.g, CollectData usaxs.mac) 2.b. if “ AutomaticCollectionStringInput” = “preUSAXStune” then : 2.b.1 set “StartAutomaticCollection” = 0 2.b.2 run command preUSAXStune 2.c. if “ AutomaticCollectionStringInput” = “useModeRadiography” then : 2.c.1 set “StartAutomaticCollection” = 0 2.c.2 run command useModeRadiography

and that basically is it (I may want to add few more commands as needed). After CollectData is done, this routine will return to sleep loop waiting for new file name or command and “StartAutomaticCollection” = 1

You get the hint here… We need users to have just the few needed options what to do and these are defined by “AutomaticCollectionStringInput” .

We need to decide what happens, if user chooses to “Stop after this scan” from the GUI. I think this command should put users back to sleep loop of AutoCollectData. This needs to be checked and somehow designed - we may need to figure out more how to do this. I see this as challenge where you can help…

We also should have some way of stopping everything (like ctrl-C) eventually, but users should be in so well “walled garden” with enough safety features, that they should be unable to screw up. And staff is responsible for rest, and we can always attach to nxsession with terminal and break what we want and care.

---

Now, there should be some niceties : Print in command line that it is running, may be also how long etc. Check the name if it is real file, may be even mac or txt file. To prevent user screwups. May be do some other meaningful checks, but most are done by our CollectData anyway, so I do not think we care too much at this time. We may add more later. I am not sure if there will be any weird surprises which spec throws at us all the time… That’s why I am asking expert.

---

As I said, I can probably write rudimentary code myself. By you are expert. Also, I would expect this exists somewhere, seems logical and trivial way to control spec - and BS - through epics and make our systems push button operated with little effort.

[prjemian]

@jkirchman Let's discuss this here so we can find it later.

[tacaswell]

We have a semi-working prototype of this use pattern: https://github.com/bluesky/bluesky-queueserver

[prjemian]

@tacaswell ; Thanks. I'll check in on that.

[jkirchman]

This is not a difficult task in SPEC either. Basically a loop with a C-like case statement. 90% can be done right away but the "niceties" which make it more useful will take time to polish. Mostly since you are never sure of what things the users would like to see and what things are beneficial to the scientist for monitoring status. It's a guessing game which will require modifications along the way.

Jan noted that he would need 2 new PVs. I'd suggest giving all of them the same prefix but that's just me. If you want more PVs later (e.g. start time, etc) then they could also start with that same prefix and it is easier to find them in your IOC: BO “AutomaticCollectionStart” Stringout “AutomaticCollectionStringInput”

[jilavsky]

I created following PVs: 9idcLAX:AutoCollectionStrInput (stringout) 9idcLAX:AutoCollectionStart (bo) znam: Stop onam: Start These are available now, teh string has "usaxs.mac" in it.

[prjemian]

This is not intuitive since neither SPEC nor Bluesky will be able to comply with 9idcLAX:AutoCollectionStart = "Start"

Better is a permit for automatic collection:

record (bo, "9idcLAX:AutoCollectionPermit") {
    field(ZNAM, "denied")
    field(ONAM, "allowed")
}

[prjemian]

also, unless there is a really good reason please stick to lower case

It's troublesome to explain the rules (they seem arbitrary) when to use upper or lower case.

[jilavsky]

It's not permit. It is "start" as instruction from user. Permits are philosophically something else. I can cases later, if we decide it is needed. But CammelCase is relatively readable for these long names.

[prjemian]

Not the CamelCase but the Start and Stop.

Exactly how can SPEC respond when the PV changes to Start?

[jilavsky]

if(PV="Start") ...start to run the script doing something else ...sleep(2) endif From user point of view he/she is directing system to Start someting. This is user meaningful name and not program meaningful name, this is PV user may be writing into. These need to be understandable to general public.
It can (and should) be aliased inside the BS to whatever is appropriate so it is understood in context of the program. I agree, that inside BS there can be line permit = epics_get("...Start", "short") or whatever BS version of this is and then BS will use this to permit the internal loop to execute. But in scope of beamline, permits are related to something else, PSS permit, BPS permit, etc. I have to live in all of these environments and with all of these communities. And everyone has their own - and different - understanding of what words mean. Waste of time solving these issues, life is too short.

[jilavsky]

Update: 9idcLAX:AutoCollectionStart: start = 1 stop = 0 not autosaved, assume it well starts as 0 9idcLAX:AutoCollectionStrInput stringout, contains whatever we put there. autosaved

[prjemian]

You did not answer this question. Exactly how can SPEC respond when the PV changes to Start?

[prjemian]

The bluesky-queueserver provides a generic scanning service with a web server interface. This request is for similar but triggered by EPICS PV, not a POST message to a web server. We'll keep that mind as we continue with an EPICS-based service that re-uses components from our existing interface to run batch of scans from command list.

[prjemian]

Part of the motivation to continue with our existing service is to limit the set of operations provided to the remote user. This part is already in place.

[prjemian]

Ah, so (as you showed with SPEC code)[https://github.com/APS-USAXS/ipython-usaxs/issues/339#issuecomment-638972814}, the command line is not sitting idle. Rather, it is running the macro for remote ops. Thus, the Start PV is the trigger to start the user's command sequence.

I agree that's different than the Permit PV I described which tells the remote ops macro to exit.

Similar in bluesky then:

RE(remote_ops())

which monitors the Start PV to trigger a collection to start, taking direction from 9idcLAX:AutoCollectionStrInput. Is that string PV a file name, such as usaxs.mac with text content which can be handled by run_command_file?

We may still want to use a waveform string to allow a longer file path to be given. Agree?

[prjemian]

To enable this feature (stated above):

• automated plan can be exited by ^C from console or by special command in the trigger (or other) PV

the Permit PV is needed.

When remote_ops() starts, it sets the permit to enable.

[jilavsky]

I can change 9idcLAX:AutoCollectionStrInput to waveform, I just need to find syntax for it. From user point of view waveform is just longer string, so that is fine. Even though, my hope is this will be short string, up to 20-30 characters long at most. But who knows...

I can add any PVs BS implementation requires. These can be named as suitable for BS, provide list and type (or just add them to LAX and reboot). I am not 100% sure I understand what the permit PV will be for in BS, but I see it is useful for GUI to display message to user that remote operations are enabled. That is something I did not consider and will be really useful for users.

hm... thinking about it... I did not expect to have to abort automated plan using epics, but for BS it may be preferred way to avoid having to deal with RE.abort stuff... Kind of graceful end to the automatic data collection - same as we engineered into our regular data collection sequence. End when sensible, I guess? There the name permit seems sensible.

[prjemian]

Something like this (note the Start PV goes back to zero after the StrInput has been executed. It signals that the process is done.

Here's the basics of a bluesky plan and ophyd object for this:

from bluesky import plan_stubs as bps
from ophyd import Component, Device, EpicsSignal, EpicsSignalRO

class AutoCollectDataDevice(Device):
    acquire = Component("Start")
    commands = Component("StrInput")
    permit = Component("Permit")
    idle_interval = 2       # seconds

    def remote_ops(self, *args, **kwargs):
        yield from bps.mv(self.permit, "enable")
        logger.info("waiting for user commands")
        while self.permit.get() in (1, "enable"):
            if self.acquire.get() in (1, "start"):
                logger.debug("starting user commands")
                command = self.commands.get()
                if command == "preUSAXStune":
                    yield from preUSAXStune()
                elif command == "useModeRadiography":
                    yield from useModeRadiography()
                elif os.path.exists(command):
                    yield from run_command_file(command)
                else:
                    logger.warning("unrecognized command: %s", command)
                yield from bps.mv(self.acquire, 0)
                logger.info("waiting for next user command")
            else:
                yield from bps.sleep(self.idle_interval)

auto_collect = AutoCollectDataDevice("9idcLAX:AutoCollection", name="auto_collect")

[prjemian]

I'll dig up that syntax after the next telecon...

[prjemian]

Probably should also have a PV that shows the auto_collect plan is running. At 8-ID-I, we use a longout record and update it at about 10 Hz in a separate thread from the ophyd object. If the PV updates at ~10 Hz, it is running. If not updating, then not running. If much faster, look for multiple processes running. We should do that here, as well. Needs another PV.

[prjemian]

that PV increments at 10 Hz and wraps to 0 at 10^4 or 10^5

[prjemian]

from direct from 8-ID-I:

    def incrementTicker(self):
        """increment the ticker to show process is working"""
        n = max(int(self.registers.workflow_ticker.get()), 0)
        self.registers.workflow_ticker.put((n + 1) % self.increment_modulo)

This needs to run in its own thread so it it not delayed by user command execution. Also the remote_ops code needs to start this thread and the thread needs to recognize to kill itself when the remote_ops() terminates (and/or the RunEngine stops running).

[prjemian]

waveform string example:

record(waveform, "demo:1kB_string") {
    field(FTVL, "CHAR")
    field(NELM, "1024")
}

[prjemian]

To begin remote operations:

RE(auto_collect.remote_ops())

[prjemian]

I tend to prefer the graceful shutdown over the abrupt one so we can have extra features start/stop with some control. It is brutal to ^C out. This could, for example, leave background threads running such as the incrementer thread (idea from 8-ID-I) that says remote_ops is active.

[jilavsky]

There already is PV stating instrument is running, Start PV should go back to zero BEFORE the StrInput has been executed. To prevent any chances of trying to run it second time. If it fails inside the execution, it fails and we should not try to run it second time, which would happen if we left the Start PV as 1. Set it to 0 as soon as we get inside the execution loop.

I changed the 9idcLAX:AutoCollectionStrInput in waveform and rebooted VME. It should be available.

We need PV which shows the AutoCollect is running, but that is all. We have other PVs telling us, that we are collecting data. It is controlling the GUI message and triggers the music after scan is done. There is lot of support already in there, if we can reuse existing code running manually.

[prjemian]

@jilavsky Look at changeset 48e5bb6

[prjemian]

@jilavsky commented in related issue:

this looks correct or at least close. One issue we have with spec implementation. The code which is used to stop collecting data (the one which reacts on StopAfterNextScan or whatever the name of that PV is) will stop not only dataq collection, btu also this Autocollect loop. That is problem, as stopping collection that way should ideally return users to this wait llop and abort altogether. How will BS implementation react to this event?

[prjemian]

These are the general terms: https://github.com/APS-USAXS/ipython-usaxs/blob/03fe162a3cf6099e880ae51a3a47c95da2bf24e0/profile_bluesky/startup/instrument/devices/general_terms.py#L272-L273

That term should be handled here but the plan is marked TODO:: https://github.com/APS-USAXS/ipython-usaxs/blob/03fe162a3cf6099e880ae51a3a47c95da2bf24e0/profile_bluesky/startup/instrument/plans/requested_stop.py#L16

Looks like we got interrupted. Right?

[prjemian]

More so, the entire plans directory looks incomplete (at least on GitHub). I seem to remember this work was interrupted for a different project.

[prjemian]

To answer your question, bluesky should not exit the RunEngine when StopBeforeNextScan goes True. It should exit the run sequence. The while loop may need to include a try .. except .. clause to handle a custom exception for this situation.

SPEC macros could be modified to watch for a special global so that the outermost loop recognizes the same situation rather than just quitting the macro execution. Might need to be creative with this.

[prjemian]

With the old code (note it is in the archive), the RunEngine will exit: https://github.com/APS-USAXS/ipython-usaxs/blob/48e5bb60380b0dbcf47ddf8ecd6d9159cc8fdad0/profile_bluesky/startup/archive/41-commands.py#L133

For sure will need that try .. except RequestAbort .. clause to trap this.

[jilavsky]

Yes, this is head scratcher in spec also. In Igor I would simply enumerate the stack of functions and decide which ones to break up to. Here, I am not sure I know what to do. And in spec Jeff is scratching his head for few days. This - and rest of BS code - can be tested on Wednesday when we restart. We will need to wait for some time to warm up things anyway. I should be there.

[prjemian]

Ok, need to check what's on usaxscontrol before proceeding.

[prjemian]

Here is the current line of code that raises the exception: https://github.com/APS-USAXS/ipython-usaxs/blob/48e5bb60380b0dbcf47ddf8ecd6d9159cc8fdad0/profile_bluesky/startup/instrument/plans/requested_stop.py#L56

So, the only remaining work is to handle this exception in the while loop with try .. except ..

[prjemian]

@jilavsky : We could have tested this code this week. Let's make sure we test it next time. Do we have all the PVs we need? Think so.

[prjemian]

Just tested this at the instrument. ! Success! Also, tried Stop before next scan and this did not interrupt the auto_collect either. Declaring victory.