sample mixing system

[prjemian]

@jilavsky emailed:

Peter Beaucage is bringing his own sample mixing system which will need to be integrated with our USAXS somehow. We need to send some signals between USAXS (spec or BlueSky if it is running at that time) and his device to coordinate data collection.

Makes me wonder if this is great opportunity to utilize our remote operations epics controls we have.

His experiment is scheduled to start after 2 day shutdown (6/27 and 6/28), so there is time to get stuff setup and tested. It is 3 days experiment, so we really need this automated.

[pbeaucage]

Thanks! More info on the system here: https://www.nist.gov/ncnr/ncnr-facility-upgrades/autonomous-formulation-lab-afl

We've done the interfacing in a few different ways at this point - from directly inserting commands into a spec session to a semaphore system in EPICS (we're actually developing this in the next couple days for an expt at ISIS). Generally speaking we scrape the data off of disk though I have a small project that sits in EPICS and reduces areadetector data (https://github.com/pbeaucage/EpicsADLiveProcess), which is a spin-out of the first generation data retrieval strategy.

At the simplest level, we just need a Python command we can call to trigger a measurement with a given sample name. For USAXS, I'd think we would want a full USAXS/SAXS/maybe WAXS sweep. We have an API for this in our internal codebase, but typically we write a small wrapper class for each instrument we work with, so we can work on practically any API.

Level 2 is that we would ideally borrow the basic Python scan reduction code @prjemian developed years ago. I think real USAXS reduction in Python is probably beyond the scale we can tackle for this experiment.

We have a control laptop (standard NIST locked-down Linux) which we can send ahead for an APS IT inspection so that we can get it on the instrument network.

Tagging @martintb from our side

[pbeaucage]

It may be hard to follow, but this is an example of how we interface with a SAXS instrument running spec at CHESS:

def expose(self,name=None,exposure=None,nexp=1,block=True,reduce_data=True,measure_transmission=True,save_nexus=True):
        if name is None:
            name=self.getFilename()
        else:
            self.setFilename(name)

        if exposure is None:
            exposure=self.getExposure()
        else:
            self.setExposure(exposure)

        self.status_txt = f'Starting {exposure} s count named {name}'
        if self.app is not None:
            self.app.logger.debug(f'Starting exposure with name {name} for {exposure} s')

        pos_trans = []
        pos_raw_data = []
        pos_simple_trans = []
        pos_reduced_data = []

        for idx,pos in enumerate(self.config['measurement_positions']):
            self.client.run_cmd(f'umv samx {pos}')
            self.client.run_cmd(f'opens')
            self.client.run_cmd(f'tseries {nexp} {exposure/10}')
            self.client.run_cmd(f'tseries {nexp} {exposure}',block=True)
            self.client.run_cmd(f'closes')

            pos_raw_data.append(self.getData())
            pos_reduced_data.append(self.getReducedData(write_data=True,filename=f'{name}_{idx}'))
            pos_trans.append(self.lastTransmission(return_full=True))
            pos_simple_trans.append(pos_trans[-1][0])

        #time.sleep(0.5)
        if block or reduce_data or save_nexus:
            self.client.block_for_ready()
            self.status_txt = 'Accessing Image'
            self.app.logger.debug(f'Min transmission was {np.min(pos_simple_trans)} at index {np.argmin(pos_simple_trans)}; returning that data.  Other transmissions were {pos_simple_trans}')
            data = pos_raw_data[np.argmin(pos_simple_trans)]
            transmission = pos_trans[np.argmin(pos_simple_trans)]
            #transmission = self.lastTransmission(return_full=True)
            if save_nexus:
                self.status_txt = 'Writing Nexus'
                self._writeNexus(data,name,name,transmission)
            if reduce_data:
                self.status_txt = 'Reducing Data'
                reduced = pos_reduced_data[np.argmin(pos_simple_trans)]
                print(np.shape(reduced))
                np.savetxt(f'{name}_chosen_r1d.csv',np.transpose(reduced),delimiter=',')
                #self.getReducedData(write_data=True,filename=name)
                #if save_nexus:
                    #self._appendReducedToNexus(reduced,name,name)
            self.status_txt = 'Instrument Idle'
            return transmission

 @Driver.unqueued(render_hint='2d_img',log_image=True)
    def getData(self,**kwargs):
        try:
            filepath = self.getLastFilePath()
            data = np.array(PIL.Image.open(filepath))
        except FileNotFoundError:
            nattempts = 1
            while nattempts<11:
                nattempts = nattempts +1
                time.sleep(0.2)
                try:
                    filepath = self.getLastFilePath()
                    data = np.array(PIL.Image.open(filepath))
                except FileNotFoundError:
                    if nattempts == 10:
                        raise FileNotFoundError(f'could not locate file after {nattempts} tries')
                    else:
                        warnings.warn(f'failed to load file, trying again, this is try {nattempts}')
                else:
                    break

        return np.nan_to_num(data)

This all sits inside a rather large codebase at this point that makes these functions callable and queued over the web, renders the output in a browser-friendly way, etc. getData pipes into a PyFAI-backed reduction system so that it's turned into reduced data. Of course that is useless for USAXS - but an example of how we approach it for 2D area detector instruments.

[pbeaucage]

A more minimal example of what I mean by EPICS semaphores: on the LARMOR SANS instrument at ISIS, there's a PV that we can set to 1 to trigger a run in their counter system. We can then watch another PV for when the run is complete and grab the data from the filesystem and reduce it.

import epics as pye

def beginrun():

    rstate=pye.caget("IN:LARMOR:DAE:RUNSTATE")

    if rstate == 1:

        pye.caput("IN:LARMOR:DAE:BEGINRUN",1)

    while rstate != 2:

        rstate=pye.caget("IN:LARMOR:DAE:RUNSTATE")

        time.sleep(2)

def getRunTitle():

    title=(pye.caget("IN:LARMOR:DAE:TITLE"))

    stitle=""

    for i in title:

        stitle+=chr(i)

    return stitle

def waitforframes(frames=5000):

    print("Waiting for {} frames".format(frames))

    frs=pye.caget("IN:LARMOR:DAE:GOODFRAMES")

    while frs < frames:

        frs=pye.caget("IN:LARMOR:DAE:GOODFRAMES")

    print("{} frames counted".format(frames))

print(pye.caget("IN:LARMOR:DAE:IRUNNUMBER"))

print(pye.caget("IN:LARMOR:DAE:INSTNAME"))

pye.caput("IN:LARMOR:DAE:TITLE:SP","Testing pyEPICS")

beginrun()

waitforframes(100)

pye.caput("IN:LARMOR:DAE:ABORTRUN",1)

print(pye.caget("IN:LARMOR:DAE:GOODUAH"))

print(pye.caget("IN:LARMOR:DAE:GOODFRAMES"))

#pye.caput("IN:LARMOR:DAE:ENDRUN",1)

[jilavsky]

There may be better solution which Pete might suggest, but in order to support remote operations we have few PVs which enable epics controls. We have PV which has command or name of command file written in working directory (for spec of BlueSky) and we have PV which tells instrument to run the command or execute the command file. There is also PV which tells us if instrument is done running. The dumbest solution would be to write the command file on LARMOR and sftp it to proper location, set the execute PV, wait for the "Running" PV to tell us it is done and then grab via sftp the files, reduce and evaluate... We need to figure out how to pass back the final name of the collected files, but that could be scraped from the scanrecord which is on the web. Ouch... seems cumbersome. This will work if the control notebook is on internal network where it has access to PVs and can reach usaxscontrol.

[pbeaucage]

This would work well, it's basically what we are doing to interface with the SANS here at ISIS.

The only parameter we really change is the filename, so it could just be a fixed script that looks in a PV (hosted in a small caproto ioc or just a string somewhere on an existing IOC) for that filename. The rest of the signaling sounds very straightforward.

sftp is a little painful to get the files, but would work. We can also have a background rsync process constantly mirror the directory over to the control notebook.

[jilavsky]

rsync would work better than sftp, but I am not sure if we can nfs mount the needed drives. If yes, we can even read directly from data drive. I assume there are better solutions than what I propose, but what I propose exists already.

[prjemian]

rsync is the likely answer, the authentication would be simpler if the laptop is on the controls subnet

[pbeaucage]

@prjemian (or maybe @jilavsky) what's involved in getting the laptop onto the controls subnet? It's a Dell laptop with an eGPU running Ubuntu 20.04LTS with a NIST management config on it. We have sudo though. We can send one ahead of the rest of the hardware if IT needs to look over it.

[jilavsky]

Question for IT. I will ask.