reduced 1-D data in NX file for uascan (step scan)

[prjemian]

Users report they cannot find reduced 1-D data in NeXus file from uascan step scans.

Are we not saving the 1D data in step scan Nexus file? It plots on the livedata web page.

Thanks to Peter Beaucage, @jilavsky for spotting this.

[prjemian]

Apparently not: https://github.com/APS-USAXS/usaxs-bluesky/blob/ae08635e4b107832a1cc7a35474592a97d7c471a/instrument/callbacks/nxwriter_usaxs.py#L3

[prjemian]

Yet.

[prjemian]

Example of current data file: Al7075FbMap1_10_0056.zip

[prjemian]

The livedata code reduces the 1-D data from the SPEC file:

def reduce_uascan(sds):
    '''data reduction of an uascan (in a SPEC file)

    :params obj sds: spec2nexus.spec.SpecDataFileScan object
    :returns: dictionary of title and R(Q)
    '''
    # get the raw data from the data file
    created_by_bluesky = sds.header.comments[0].startswith("Bluesky ")
    if not created_by_bluesky:  # SPEC created this data file
        wavelength        = float(sds.metadata['DCM_lambda'])
        ar                = numpy.array(sds.data['ar'])
        seconds           = numpy.array(sds.data['seconds'])
        pd                = numpy.array(sds.data['pd_counts'])
        I0                = numpy.array(sds.data['I0'])
        I0_amplifier_gain = numpy.array(sds.metadata['I0AmpGain'])
        pd_range          = numpy.array(sds.data['pd_range'], dtype=int)
        ar_center         = float(sds.metadata['arCenter'])

This is done by intent: step scans could be done by either SPEC or Bluesky and would receive equal treatment.

[prjemian]

@jilavsky : SPEC does not create a NeXus file for step scans. Shall we add the 1-D data reduction to the NeXus files from Bluesky uascans? Sounds like a yes vote.

[prjemian]

This is when the code starts to be repeated since reduce_uascan(), and the routines it uses, are in the livedata repository, not the bluesky instrument repo. Also, the livedata code is still Py2.7 while bluesky is now solidly Py3.8+. Which means the one cannot call the code from the other directly.

[prjemian]

One particular problem may come up. Since the NeXus file is written by a callback, executed while the plan is running from bluesky, we might have difficulty in writing additional content until the run is finished (and the file is closed by the callback).

First off, I've added from ..callbacks import nxwriter to the _after_scan_() method in the uascan() plan. Let's verify that import does not, itself, raise an exception, such as for circular imports. Then, can test if the nxwriter object is actually open at the time.

[prjemian]

This diagnostic, added at the same point, will inform us if we can add the reduced 1-D data to the NeXus file after the scan ends but before the close document is emitted.

        from ..callbacks import nxwriter

        print(f"nxwriter is open: {nxwriter.id.valid == 1} with {nxwriter.mode=}")

[pbeaucage]

Do we need to test this on live hardware? This afternoon would be a good time if so.

An alternative, selfish solution would be for me to import a 2to3'd version of reduce_uascan() to my AFL-automation codebase. That solves the 'today' problem and might be an acceptable long-term solution given that flyscanning will eventually come back to the instrument.

[prjemian]

@pbeaucage : Do what you need for the today problem. Long term, the livedata code will be upgraded to Py3.8+.

With the diagnostic code added today, I'm hopeful we learn how quickly we can integrate writing of reduced data. I've got time 2-4 today.

[prjemian]

With APS-U, this software should be installed as packages, in various modules, so the same code can be used in the multiple contexts (data recording, visualization, ...) of the USAXS instrument.

[prjemian]

I added a few lines of code to the uascan() plan (in instrument/plans/uascan.py lines 257~267) to log a diagnostic that informs us if the NeXus file is available for updating. Can you restart bluesky to get this additional diagnostic? It will appear on the console (lines starting with I ... DIAGNOSTIC) and in the log file.

[prjemian]

The timing could be off to write reduced data at this point because the writer() code is invoked just before the stop document is written. The code is uascan._after_scan_() might be called before the RE senses that the plan code is complete. Sounds tricky, it's just a matter of timing. It decides how we get to the raw data for the reduction at this point in the code (we might be trying to write the reduced data before the raw data).

[prjemian]

If it's a problem to reduce at this point in the plan, then we rethink the idea and write the reduced data from the nxwriter callback by adding lines here: https://github.com/APS-USAXS/usaxs-bluesky/blob/e07cb8d9689c9224d9c659125433e974e320f60d/instrument/callbacks/nxwriter_usaxs.py#L136-L142

specifically after super().writer()

[prjemian]

Adding this code in the callback, must test that it does not cause other problems (such as if this code is used by other plans).

[prjemian]

Since there is an upgrade to the NXWriter that avoids a certain deadlock (involving a special case we do not have here with a uascan), the new code will run the writer in a background thread. Short story: we should not modify the writer() method, but rather add 1-D data reduction at the end of the write_entry() method: https://github.com/APS-USAXS/usaxs-bluesky/blob/e07cb8d9689c9224d9c659125433e974e320f60d/instrument/callbacks/nxwriter_usaxs.py#L68-L77

At this point, all the data we will need has been written to the file (which we know is still open for write access). In the future, we'll need to nxwriter.wait_writer() after we run the uascan() plan before proceeding to the next thing for the RE to do.

[prjemian]

Note:

(bluesky_2023_1) bash-4.4$ git grep NXWriter
instrument/callbacks/nxwriter.py:from .nxwriter_usaxs import NXWriterUascan
instrument/callbacks/nxwriter.py:nxwriter = NXWriterUascan()
instrument/callbacks/nxwriter_usaxs.py:    # "NXWriterFlyScan",    # not yet tested
instrument/callbacks/nxwriter_usaxs.py:    "NXWriterUascan",
instrument/callbacks/nxwriter_usaxs.py:    # "NXWriterSaxsWaxs",    # not yet tested
instrument/callbacks/nxwriter_usaxs.py:from apstools.callbacks import NXWriterAPS
instrument/callbacks/nxwriter_usaxs.py:class OurCustomNXWriterBase(NXWriterAPS):
instrument/callbacks/nxwriter_usaxs.py:    customize the NXWriter for this instrument
instrument/callbacks/nxwriter_usaxs.py:    def write_monochromator(  # override NXWriterAPS
instrument/callbacks/nxwriter_usaxs.py:class NXWriterFlyScan(OurCustomNXWriterBase):
instrument/callbacks/nxwriter_usaxs.py:class NXWriterSaxsWaxs(OurCustomNXWriterBase):
instrument/callbacks/nxwriter_usaxs.py:class NXWriterUascan(OurCustomNXWriterBase):

In this issue, we focus on the NXWriterUascan class. Any changes to the write_entry() method must be within the NXWriterUascan class.

[pbeaucage]

Looks promising:

I Tue-13:20:25 - DIAGNOSTIC: this is when to write reduced 1-D data
I Tue-13:20:25 - DIAGNOSTIC: HDF5 file='/share1/USAXS_data/2023-03/02_28_Beaucage/02_28_Beaucage_usaxs/p12_D1_0025.h5'
I Tue-13:20:25 - DIAGNOSTIC: Is HDF5 file open? True
I Tue-13:20:25 - DIAGNOSTIC: HDF5 file access mode=r+

[prjemian]

I'm making progress on pulling the reduction code from livedata. Next, will need to get the code that writes it into the correct group(s).

[prjemian]

Here's a fragment of the structure of reduced Fly scan data in one of our NeXus files:

    flyScan_reduced_250:NXdata
      @NX_class = "NXdata"
      @Q_indices = 0
      @axes = "Q"
      @signal = "R"
      @timestamp = "2022-02-19 14:14:47"
      Q:NX_FLOAT64[248] = [3.3933885641557397e-06, 1.0901137072808442e-05, 1.75024709615666e-05, '...', 0.2915307210114776]
        @units = "1/A"
      R:NX_FLOAT64[248] = [1100.5120539868337, 1047.1706464423587, 970.1334645593068, '...', 5.582052895372627e-06]
        @units = "none"
      dR:NX_FLOAT64[248] = [11.482771449096276, 10.471706464423587, 26.958520621959565, '...', 3.0773888096531315e-07]
        @units = "none"
    flyScan_reduced_full:NXdata
      @NX_class = "NXdata"
      @Q_indices = 0
      @axes = "Q"
      @signal = "R"
      @timestamp = "2022-02-19 14:14:47"
      Q:NX_FLOAT64[7981] = [-0.0001642972784511932, -0.00015915800299892612, -0.00015403234258865343, '...', 0.29987013681277114]
        @units = "1/A"
      R:NX_FLOAT64[7981] = [0.1782618748791915, 0.1939469180511154, 0.21083256840168987, '...', 5.352240689734151e-06]
        @units = "none"
      R_max:NX_FLOAT64 = 1112.0547295738131
        @units = "none"
      ar:NX_FLOAT64[7981] = [8.75090914676693, 8.750881477930989, 8.750853882395724, '...', 7.1355300693797705]
        @units = "degrees"
      ar_0:NX_FLOAT64 = 8.750024602927025
        @units = "degrees"
      ar_r_peak:NX_FLOAT64 = 8.750014867089128
        @units = "degrees"
      r:NX_FLOAT64[7981] = [0.00019403805080599998, 0.000220827960893, 0.00023954796421800001, '...', 1.30862284864e-09]
      r0:NX_FLOAT64[7981] = [0.0010885, 0.0011386, 0.0011362, '...', 0.0002445]
      r_peak:NX_FLOAT64 = 1112.0547295738131
        @units = "none"

Hint: We're not using NXcanSAS here, just an ordinary NXdata group.

[prjemian]

We did NOT capture any error messages in the log files. Maybe something was shown onscreen but I did not see it when we tested. Pivot the logger statements to print for immediate diagnostics.

[pbeaucage]

generator uascan ['166d0c26'] (scan num: 347)
I Tue-15:46:11 - DIAGNOSTIC: this is when to write reduced 1-D data
I Tue-15:46:11 - DIAGNOSTIC: HDF5 file='/share1/USAXS_data/2023-03/02_28_Beaucage/02_28_Beaucage_usaxs/p9_E8_0045.h5'
I Tue-15:46:11 - DIAGNOSTIC: Is HDF5 file open? True
I Tue-15:46:11 - DIAGNOSTIC: HDF5 file access mode=r+
I Tue-15:46:11 - /home/beams/USAXS/bluesky/instrument/callbacks/calculate_reduced_data.py
I Tue-15:46:11 - TODO: save reduced uascan data to group: /entry/uascan_reduced_full

After a restart, I'm getting the TODO now, and I have uascan_reduced_full group in an example output file /mnt/usaxscontrol/USAXS_data/2023-03/02_28_Beaucage/02_28_Beaucage_waxs/p9_E8_0045.hdf

Victory?

[prjemian]

Victory!

[prjemian]

Does it suit your needs? Anything else to be done here?

[prjemian]

I see the pattern done here could also be applied to Fly Scans, in the other class in the same file.

[prjemian]

On a separate issue, that is.

[pbeaucage]

I think this covers it! The only other thing would be filtering intensity dropouts during range switches, but that's there in the webplot code too. I think I can deal with it on my end.

Thanks for the help!