fix patches for databroker 1.2.3

[ambarb]

• [ ] 1. inout in baseline reports values that are labeled as floats, not strings. The type() should be changed to string

• [x] 2. fccd_image shape is taken from top level AD setup. WRitten files should get image shape from the plugin responsibble for writing the image.

For item 2, the HDF plugin writes the file, and the PVs associated with array shape are :

XF:23ID1-ES{FCCD}HDF1:ArraySize0_RBV
XF:23ID1-ES{FCCD}HDF1:ArraySize1_RBV

The raw images is rotated in CSXtools so the user sees something different. The unrotated images has the first number being horizontal, and the second number being veritcal.

After processing by CSXtools, image shape flips axes

This is extra confusing because the raw image is not the image handled through the ROI and STATS plugins so don't change sutff associated with ROI and STATS as this represents the postCSXtools process orientation that use users sees activley in imageJ at the beamline.

[danielballan]

Item 1 is now being patched at data-access time by a "databroker transform":

# /nsls2/software/etc/tiled/profiles/csx_transforms.py
import copy

def patch_descriptor(doc):
    doc = copy.deepcopy(doc)
    # This was labeled "integer" but it is actually "string".
    INOUT_KEY = "inout_status"
    if INOUT_KEY in doc["data_keys"]:
        doc["data_keys"][INOUT_KEY]["dtype"] = "string"
    return doc

Fixing that at the source wouldn't hurt, but it's low priority.

Item 2 cannot be patched in this way, so it's currently being fixed in databroker migration that is manually run. Fixing this at the source would remove the need for the migration and enable data from CSX to be immediately readable using "secure-able databroker" (i.e. from jupyter.nsls2.bnl.gov).

[danielballan]

Notes from fixing Item 2 at the source

Before any changes:

In [20]: RE(count([fccd]))
<snipped>

In [21]: db[-1].descriptors[0]['data_keys']['fccd_image']['shape']
Out[21]: [1, 2050, 1152]

In [22]: np.asarray(list(db[-1].data('fccd_image'))).shape
Out[22]: (1, 1, 1000, 960)

In [23]: fccd.make_data_key()
Out[23]: 
{'shape': (1, 2050, 1152),
 'source': 'PV:XF:23ID1-ES{FCCD}',
 'dtype': 'array',
 'external': 'FILESTORE:'}

After:

In [1]: fccd.make_data_key()
Out[1]: 
{'shape': (1, 1000, 960),
 'source': 'PV:XF:23ID1-ES{FCCD}',
 'dtype': 'array',
 'external': 'FILESTORE:'}

In [6]: db[-1].descriptors[0]['data_keys']['fccd_image']['shape']
Out[6]: [1, 1000, 960]

In [5]: RE(count([fccd]))
<snipped>

In [7]: np.asarray(list(db[-1].data('fccd_image'))).shape
Out[7]: (1, 1, 1000, 960)

[danielballan]

The fix is https://github.com/NSLS-II-CSX/xf23id1_profiles/commit/7d74ef0c7f78e01ada597713b652fd7a20a743ec.

[danielballan]

Having fixed the way shape metadata is written, I then hand-edited /etc/databroker/csx.yml to point to the migrated11 database so that the bsui process will write to that database directly. This change will need to be applied on all hosts where bsui is run and captured in configuration control.

I also went back for the old data that we had not yet migrated (30 May to the present). All data back to 2017 should now be readable.

[danielballan]

The change was manually applied to:

  614  ssh xf23id1-ws1.nsls2.bnl.local
  615  ssh xf23id1-srv1.nsls2.bnl.local
  616  ssh xf23id1-ws4.nsls2.bnl.local
  617  ssh xf23id1-ws5.nsls2.bnl.local
  618  ssh xf23id1-ws3.nsls2.bnl.local
  619  ssh xf23id1-srv2.nsls2.bnl.local
  620  ssh xf23id1-srv3.nsls2.bnl.local
  621  ssh xf23id1-srv3.nsls2.bnl.local

because Satellite was not working, but the change has been merged into the devops repo.