Cannnot find /entry/instrument/monochromator in uascan (NeXus) file

[jilavsky]

My Igor code is now failing on step scanned USAXS files from BS. Looks to me that instrument/monochromator disappeared and now we have instrument/source. My code cannot find energy and similar parameters. See 02_15_Rabe_usaxs with AirBlank_0006 being flyscan and AirBlank_0008 being step scan. Is this correct change and permanent change (and I will fix Igor code) or is this bug and we need to fix this on BS side ?

[prjemian]

Right. Not finding NeXus structure /entry/instrument/monochromator:

/entry/instrument   OPTIONAL NXDL group in data file   not found:  in /entry/instrument/monochromator                                                                             

but finding all the right values in the bluesky streams:

/entry/instrument/bluesky/streams/baseline/monochromator_dcm_energy   OK       validItemName             strict pattern: [a-z_][a-z0-9_]*                                                                                           
/entry/instrument/bluesky/streams/baseline/monochromator_dcm_energy   OK       group in base             not defined: NXnote/monochromator_dcm_energy                                                                               
/entry/instrument/bluesky/streams/baseline/monochromator_dcm_energy   OK       known NXDL                NXdata: recognized NXDL specification                                                                                      
/entry/instrument/bluesky/streams/baseline/monochromator_dcm_energy   OK       NeXus base class          NXdata: known NeXus base class                                                                                             
/entry/instrument/bluesky/streams/baseline/monochromator_dcm_energy   OPTIONAL NXDL field in data file   not found: /entry/instrument/bluesky/streams/baseline/monochromator_dcm_energy/DATA                                        
...

Agree, something has changed. Suspicious of the procedures that create the soft links.

[prjemian]

Here's the actual value from the bluesky stream:

In [18]: root["/entry/instrument/bluesky/streams/baseline/monochromator_dcm_energy/value_end"][()]
Out[18]: 21.000047131358706

In [19]: root["/entry/instrument/bluesky/streams/baseline/monochromator_dcm_energy/value_end"].attrs["units"]
Out[19]: 'keV'

[prjemian]

This location:

/entry/instrument/source                                                                                                     OK       NXDL field in data file              found: /entry/instrument/source/energy                                                                                     

This is the content of that group (as seen by punx):

      source:NXsource
        @NX_class = "NXsource"
        @target = "/entry/instrument/source"
        current --> /entry/instrument/bluesky/streams/baseline/aps_current/value_start
        cycle --> /entry/instrument/bluesky/streams/baseline/aps_aps_cycle/value_start
        energy:NX_INT64[] = 
          @units = "GeV"
        fill_number --> /entry/instrument/bluesky/streams/baseline/aps_fill_number/value_start
        name:NX_CHAR = b'Advanced Photon Source'
          @short_name = "APS"
        probe:NX_CHAR = b'x-ray'
        type:NX_CHAR = b'Synchrotron X-ray Source'

The energy field is not right. The others are the expected links to the bluesky stream.

[prjemian]

Ah, that energy is the storage ring energy. It's OK.

[prjemian]

Not seeing a monochromator group in /entry/instrument:

In [1]: import h5py

In [2]: root = h5py.File("AirBlank_0008.h5", "r")

In [3]: list(root["/entry/instrument"])
Out[3]: ['bluesky', 'slits', 'source']

yet the documentation says to expect it (see line 46 here): https://github.com/APS-USAXS/ipython-usaxs/blob/ffa78c33726e6fc470ab1c5ff42f88a51c526281/profile_bluesky/startup/instrument/callbacks/nxwriter_usaxs.py#L38-L47

[prjemian]

I suspect that the schema for the bluesky streams has changed and this writer was not yet updated or tested. (Note that /entry/instrument/bluesky/streams exists now but line 44 (above) says /entry/instrument/bluesky_streams.

The change in the schema allows more data (all streams, not just a select few) from bluesky to be recorded:

In [4]: list(root["/entry/instrument/bluesky"])
Out[4]: ['metadata', 'plan_name', 'streams', 'uid']

In [5]: list(root["/entry/instrument/bluesky/streams"])
Out[5]: ['aps_current_monitor', 'baseline', 'primary']

[prjemian]

Looking at apstools.callbacks.nexus_writer.NXWriter.get_stream_link() to see if we need to supply additional arguments when we call it now...

[prjemian]

Created callable function from the get_stream_link() method to test without the NXWriter class itself:

def get_stream_link(root, signal, stream=None, ref=None):
    """
    return the h5 object for ``signal``
    DEFAULTS
    ``stream`` : ``baseline``
    ``key`` : ``value_start``
    """
    stream = stream or "baseline"
    ref = ref or "value_start"
    h5_addr = (
        f"/entry/instrument/bluesky/streams/{stream}/{signal}/{ref}"
    )
    if h5_addr not in root:
        raise KeyError(f"HDF5 address {h5_addr} not found.")
    # return the h5 object, to make a link
    return root[h5_addr]

and then call it with one of the lines from instrument.callbacks.nxwriter_usaxs.py:

In [9]: get_stream_link(root, "user_data_sample_title")
Out[9]: <HDF5 dataset "value_start": shape (), type "|O">

In [10]: get_stream_link(root, "user_data_sample_title")[()]
Out[10]: b'AirBlank'

In [12]: get_stream_link(root, "user_data_sample_title")[()].decode()
Out[12]: 'AirBlank'

[prjemian]

uascan file saving needs to be update to current apstools support.

Where is the monochromator group saved?

[prjemian]

See here: https://github.com/BCDA-APS/apstools/blob/3722eff8cf205ea03e274ac775df963f9ae2b847/apstools/callbacks/nexus_writer.py#L380

[prjemian]

Here is the stream value using the test function:

In [18]: get_stream_link(root, "monochromator_dcm_energy")[()]
Out[18]: 21.000047131358706

[prjemian]

Data exists but is not linked. That's what needs to be fixed. Might need to override (?and restore from our previous local code?) from apstools support.

[prjemian]

In apstools, the monochromator group expects to find these keys:

keys = "wavelength energy theta y_offset mode".split()

but we don't have the y_offset key in our baseline stream (also, don't have mode):

In [25]: [k for k in nm if k.find("offset") >= 0]
Out[25]: 
['terms_SAXS_ax_limit_offset',
 'terms_SAXS_dx_limit_offset',
 'terms_SAXS_y_limit_offset',
 'terms_SAXS_z_limit_offset',
 'terms_USAXS_start_offset',
 'terms_WAXS_x_limit_offset']

In [26]: [k for k in nm if k.find("monochromator") >= 0]
Out[26]: 
['monochromator_cryo_level',
 'monochromator_dcm_energy',
 'monochromator_dcm_energy_setpoint',
 'monochromator_dcm_theta',
 'monochromator_dcm_theta_setpoint',
 'monochromator_dcm_wavelength',
 'monochromator_dcm_wavelength_setpoint',
 'monochromator_dcm_y1',
 'monochromator_dcm_z2',
 'monochromator_feedback_control',
 'monochromator_feedback_drvh',
 'monochromator_feedback_drvl',
 'monochromator_feedback_on',
 'monochromator_feedback_oval',
 'monochromator_temperature']

Could add this signal to our baseline or make a local override of write_monochromator() that picks different keys. (Could add keys kwarg to the apstools support in a future update; that would be more flexible).

Going with the local override for now.

[prjemian]

The logs are not informative to any possible error information:

|2022-02-15 08:44:42.333|INFO|32635|bluesky-session|nxwriter|15|MainThread| - /home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/callbacks/nxwriter.py
|2022-02-15 08:44:42.334|INFO|32635|bluesky-session|nxwriter_usaxs|16|MainThread| - /home/beams/USAXS/.ipython/profile_bluesky/startup/instrument/callbacks/nxwriter_usaxs.py

[prjemian]

@jilavsky : This code is in place and ready to test, when you have time available.

[jilavsky]

Sunday afternoon. I'll let you know.

[prjemian]

Links can be added in the existing files if you wish. We could try that first on a copy of the file to be modified.

[jilavsky]

These were purely test scans, no need for the data. I was simply testing, that all still works. You see the trust level I have by now ;-)

[jilavsky]

Tested, works, Igor data reduction is happy.