search

APS-USAXS / livedata

live data from the APS USAXS instrument
https://usaxslive.xray.aps.anl.gov/
0 stars 0 forks source link

Dexela support for live data page #51

Closed jilavsky closed 2 years ago

jilavsky commented 3 years ago

At this time Dexela detector data are not reduced and posted to live data graph. It is using same Nexus as Pilatus, but there are small differences. Sounds like code needs to be modified to handle also Dexela.

jilavsky commented 2 years ago

Recent test proven usefulness of this detector and future Linkam experiments from some users should be using it for regular work. Users expressed need for the web support with this detector. We have plenty of data now available.

prjemian commented 2 years ago

PV prefix for the Dexela?

jilavsky commented 2 years ago

9idcDEX, it is off line now. It was used until Sunday morning only, Sunday users needed Pilatus again.

jilavsky commented 2 years ago

Test data: USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs, any image there. Same metadata as Pilatus, should be proper Nexus file.

prjemian commented 2 years ago

@jilavsky Is the Dexela image transposed (as the Pilatus detectors)? https://github.com/APS-USAXS/livedata/blob/25113a71346f6395b03b1c40276cd36b31214a23/reduceAreaDetector.py#L46-L48

prjemian commented 2 years ago

Need to have this set as in the other detectors (it is empty in the Dexela's HDF5 files):

        x_pixel_size:NX_FLOAT64[] = 
        y_pixel_size:NX_FLOAT64[] =
jilavsky commented 2 years ago

That is confusing, I checked random file with HDFView and it has proper value there (0.0748mm). q It should, this is in layout:

  <dataset name="x_pixel_size"   value="0.0748"  source="constant" type="float"/>
  <dataset name="y_pixel_size"   value="0.0748"  source="constant" type="float"/>

Check again, please and let me know where the problem is.

prjemian commented 2 years ago

What is the HDF5 address to that value you saw in HDFView?

prjemian commented 2 years ago

These lines define the configuration that will be used:

        'x_pixel_size_mm'       : '/entry/instrument/detector/x_pixel_size',
        'y_pixel_size_mm'       : '/entry/instrument/detector/y_pixel_size',
prjemian commented 2 years ago

Aha! That information is smoewhere else:

bash-4.2$ punx tree 2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ti64Dex_975C_244min_000437.hdf | grep -i pixel
      waxs_ccd_center_x_pixel:NX_FLOAT64 = 83.18774884799959
        @NDAttrName = waxs_ccd_center_x_pixel
        @NDAttrDescription = horizontal position of beam center on CCD, pixels
      waxs_ccd_center_y_pixel:NX_FLOAT64 = 1539.139546368824
        @NDAttrName = waxs_ccd_center_y_pixel
        @NDAttrDescription = vertical position of beam center on CCD, pixels
      waxs_ccd_pixel_size_x:NX_FLOAT64 = 0.0748
        @NDAttrName = waxs_ccd_pixel_size_x
        @NDAttrDescription = CCD pixel size, horizontal, mm
      waxs_ccd_pixel_size_y:NX_FLOAT64 = 0.0748
        @NDAttrName = waxs_ccd_pixel_size_y
        @NDAttrDescription = CCD pixel size, vertical, mm
          @NDAttrName = waxs_ccd_center_x_pixel
          @NDAttrDescription = horizontal position of beam center on CCD, pixels
          @NDAttrName = waxs_ccd_center_y_pixel
          @NDAttrDescription = vertical position of beam center on CCD, pixels
        x_pixel_size:NX_FLOAT64[] = 
        y_pixel_size:NX_FLOAT64[] =

Using h5dump, we can see that links are missing in this file:

bash-4.2$ h5dump -n 2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ti64Dex_975C_244min_000437.hdf | grep pixel
 dataset    /entry/Metadata/waxs_ccd_center_x_pixel
 dataset    /entry/Metadata/waxs_ccd_center_y_pixel
 dataset    /entry/Metadata/waxs_ccd_pixel_size_x
 dataset    /entry/Metadata/waxs_ccd_pixel_size_y
 dataset    /entry/instrument/detector/beam_center_x -> /entry/Metadata/waxs_ccd_center_x_pixel
 dataset    /entry/instrument/detector/beam_center_y -> /entry/Metadata/waxs_ccd_center_y_pixel
 dataset    /entry/instrument/detector/x_pixel_size
 dataset    /entry/instrument/detector/y_pixel_size
prjemian commented 2 years ago

Looking at how the EPICS AreaDetector configures these files:

bash-4.2$ pwd
/share1/AreaDetectorConfig
bash-4.2$ git grep -i link | grep -vi linkam | grep pixel
Old/SAXS/layout_saxs.xml:        <hardlink name="beam_center_x" target="/entry/Metadata/pin_ccd_center_x_pixel"/>
Old/SAXS/layout_saxs.xml:        <hardlink name="beam_center_y" target="/entry/Metadata/pin_ccd_center_y_pixel"/>
Old/SAXS/layout_saxs.xml:        <hardlink name="x_pixel_size" target="/entry/Metadata/pin_ccd_pixel_size_x"/>
Old/SAXS/layout_saxs.xml:        <hardlink name="y_pixel_size" target="/entry/Metadata/pin_ccd_pixel_size_y"/>
SAXS_config/hdf5_plugin/layout.xml:        <hardlink name="beam_center_x" target="/entry/Metadata/pin_ccd_center_x_pixel"/>
SAXS_config/hdf5_plugin/layout.xml:        <hardlink name="beam_center_y" target="/entry/Metadata/pin_ccd_center_y_pixel"/>
SAXS_config/hdf5_plugin/layout.xml:        <hardlink name="x_pixel_size" target="/entry/Metadata/pin_ccd_pixel_size_x"/>
SAXS_config/hdf5_plugin/layout.xml:        <hardlink name="y_pixel_size" target="/entry/Metadata/pin_ccd_pixel_size_y"/>
WAXS_config/hdf5_plugin/layout_waxs.xml:        <hardlink name="beam_center_x" target="/entry/Metadata/waxs_ccd_center_x_pixel"/>
WAXS_config/hdf5_plugin/layout_waxs.xml:        <hardlink name="beam_center_y" target="/entry/Metadata/waxs_ccd_center_y_pixel"/>
WAXS_config/hdf5_plugin/layout_waxs.xml:        <hardlink name="x_pixel_size" target="/entry/Metadata/waxs_ccd_pixel_size_x"/>
WAXS_config/hdf5_plugin/layout_waxs.xml:        <hardlink name="y_pixel_size" target="/entry/Metadata/waxs_ccd_pixel_size_y"/>

no such configuration for the Dexela detector.

bash-4.2$ git grep hardlink | grep -i dexela
bash-4.2$
prjemian commented 2 years ago

Just this:

bash-4.2$ grep -i pixel Dexela/layout_waxs.xml 
        <hardlink name="beam_center_x" target="/entry/Metadata/waxs_ccd_center_x_pixel"/>
        <hardlink name="beam_center_y" target="/entry/Metadata/waxs_ccd_center_y_pixel"/>
        <dataset name="x_pixel_size"   value="0.0748"  source="constant" type="float"/>
        <dataset name="y_pixel_size"   value="0.0748"  source="constant" type="float"/>
bash-4.2$

h5dump's view:

            DATASET "x_pixel_size" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SCALAR
               DATA {
               (0): 0.0748
               }
            }
            DATASET "y_pixel_size" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SCALAR
               DATA {
               (0): 0.0748
               }
            }

punx's view:

        x_pixel_size:NX_FLOAT64[] = 
        y_pixel_size:NX_FLOAT64[] =

h5py's view

In [1]: import h5py

In [2]: f = h5py.File("2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ti64Dex_975C_244min_000437.hdf", "r")

In [3]: g = f["/entry/instrument/detector"]

In [4]: "x_pixel_size" in g
Out[4]: True

In [5]: g["x_pixel_size"]
Out[5]: <HDF5 dataset "x_pixel_size": shape (), type "<f8">

In [6]: g["y_pixel_size"]
Out[6]: <HDF5 dataset "y_pixel_size": shape (), type "<f8">
prjemian commented 2 years ago

In summary, the declaration of a constant float in the layout.xml file is not working. These two are the only such declarations in that file, all the other constants are string.

bash-4.2$ grep constant ./Dexela/layout_waxs.xml | grep float
        <dataset name="x_pixel_size"   value="0.0748"  source="constant" type="float"/>
        <dataset name="y_pixel_size"   value="0.0748"  source="constant" type="float"/>
bash-4.2$ grep constant ./Dexela/layout_waxs.xml | head -5
    <attribute name="NX_class"      value="NXentry" source="constant" type="string"/>
    <attribute name="default"       value="data"    source="constant" type="string"/>
    <dataset  name="definition"     value="NXsas" source="constant" type="string"/>
    <dataset  name="program_name"   value="EPICS areaDetector" source="constant" type="string"/>
      <attribute name="NX_class" value="NXmonitor" source="constant" type="string"/>
prjemian commented 2 years ago

Investigating further, these are written to the HDF5 file as scalar values rather than arrays (like absolutely all the other numerical data in NeXus HDF5 files ever is written - assumed to be some sort of rule).

Now, we can see some sort of difference here:

In [16]: g["beam_center_x"]
Out[16]: <HDF5 dataset "beam_center_x": shape (1,), type "<f8">

In [17]: g["x_pixel_size"]
Out[17]: <HDF5 dataset "x_pixel_size": shape (), type "<f8">

In [18]: g["beam_center_x"].dtype
Out[18]: dtype('float64')

In [19]: g["x_pixel_size"].dtype
Out[19]: dtype('float64')

In [20]: g["beam_center_x"].shape
Out[20]: (1,)

In [21]: g["x_pixel_size"].shape
Out[21]: ()

which means a new rule to access scalar values in HDF5 files:

if dataset.shape == (1, ):  # historical representation of scalar value
    value = dataset[0]
elif  dataset.shape == ():  # scalar representation
    value = dataset.value
else:  # array of some sort
    value = numpy.array(dataset)
prjemian commented 2 years ago

Moving ahead as if the HDF5 files (and associated XML files used to create them) do not need to be changed, the livedata/reduceAreaDetector.py will be changed for this new development.

prjemian commented 2 years ago

@jilavsky - the livedata/reduceAreaDetector.py code runs without failing on the USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ti64Dex_975C_99min_000399.hdf file.

prjemian commented 2 years ago

I'm trying to force the Dexela data from USAXS_data/2021-12/12_10_DexHeater.dat to plot.

prjemian commented 2 years ago

Plots are not getting updated because of too many digits in when AD writes the file name:

bash-4.2$ ll /share1/USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/Ag*
-rw-rw-r-- 1 usaxs 24519664 Dec 10 17:22 /share1/USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/AgBehenateLaB6_000026.hdf
-rw-rw-r-- 1 usaxs    37497 Dec 10 17:21 /share1/USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/AgBehenateLaB6_0026.jpg
bash-4.2$ ll '/share1/USAXS_data/2021-12/12_10_DexHeater/12_10_DexHeater_waxs/AgBehenateLaB6_0026.hdf'

SPEC writes the name of the HDF5 file on the command line. With %04d format for the USAXS data number. The JPEG detector also writes its image file with %04d format. It is clear the Dexela IOC writes its image file with %06d format resulting in a file not found and no plot of the Dexela image.

prjemian commented 2 years ago

From the Dexela's bluesky setup:

    det.hdf1.stage_sigs["file_template"] = "%s%s_%6.6d.h5"

When the file name is written by SPEC on the command line, it should agree with the formatting of 9idcDEX:HDF1:FileTemplate PV.

jilavsky commented 2 years ago

I fixed spec to use 6 digit formatting. This was oversight. With template, looks like I messed up something. I thought I did minimum changes from Pilatus configuration. Dexela configuration are attributes.xml and layout.xml in AreaDetectorConfig/Dexela on share1. If these need to be fixed, work with me.

prjemian commented 2 years ago

Try it with just the change to the FileTemplate PV. Leave the XML files as-is for now.

If we really want to plot the Dexela WAXS data from Dec 10, we can either edit the SPEC file (my choice) or make soft links to the correct image files.

On Wed, Dec 15, 2021, 9:25 AM Jan Ilavsky @.***> wrote:

I fixed spec to use 6 digit formatting. This was oversight. With template, looks like I messed up something. I thought I did minimum changes from Pilatus configuration. Dexela configuration are attributes.xml and layout.xml in AreaDetectorConfig/Dexela on share1. If these need to be fixed, work with me.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/APS-USAXS/livedata/issues/51#issuecomment-994899097, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARMUMCNKBKEFZUTVY2AJRTURCXNVANCNFSM46BY65EQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

jilavsky commented 2 years ago

We will not run Dexela in December again. I edited spec to make sure it works next time = February.

prjemian commented 2 years ago

Can you test it after the run ends to verify it works as expected?