BCDA-APS / bluesky_training

Bluesky training, including instrument package
https://bcda-aps.github.io/bluesky_training/
Other
11 stars 0 forks source link

adsimdet: compress HDF5 images #237

Closed prjemian closed 1 year ago

prjemian commented 1 year ago

Set compression for HDF5 file images in the adsimdet device. Either after this line: https://github.com/BCDA-APS/bluesky_training/blob/3cc523e3613f467066601736305a78e528517da6/bluesky/instrument/devices/area_detector.py#L152 or this line: https://github.com/BCDA-APS/bluesky_training/blob/3cc523e3613f467066601736305a78e528517da6/bluesky/instrument/devices/area_detector.py#L160

prjemian commented 1 year ago

compare file sizes with different compression settings (all are 1k x 1k, 100 frames, 8 bit, mono, simulated beam spot)

compression Run time file size file name
LZ4 0.700 1.7M 5512ecb6-0b17-40c3-8fae_000000.h5
None 0.690 102M d69c1e88-95d1-4a73-858e_000000.h5
Blosc 0.657 1.9M 32227a0c-bbf3-4f9e-a6a9_000000.h5
BSLZ4 0.715 2.3M 6245e7d3-46fc-4e69-9927_000000.h5
zlib 3.488 13M 1df8c934-6534-4ffb-99d4_000000.h5

All other HDF5 plugin settings left at defaults for these measurements.

image

prjemian commented 1 year ago

Use stage_sigs since that will avoid modifying the IOC's settings when a bluesky session is started.

prjemian commented 1 year ago

For maximum client compatibility (client needs plugin support to read advanced compression methods), configure zlib as the default. It's slower than None yet 6x smaller file size.

prjemian commented 1 year ago

Note that when adding an item to hdf1.stage_sigs, keep capture as the last one.

prjemian commented 1 year ago

Here's the full exception trace when punx cannot read the file:

(bluesky_2023_3) prjemian@zap:~$ punx tree /tmp/docker_ioc/iocad/tmp/adsimdet/2023/08/22/765f4a04-0a69-4a40-89d0_000000.h5

!!! WARNING: this program is not ready for distribution.

Traceback (most recent call last):
  File "/home/prjemian/.conda/envs/bluesky_2023_3/bin/punx", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/main.py", line 414, in main
    args.func(args)
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/main.py", line 183, in func_tree
    report = mc.report(args.show_attributes)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/h5tree.py", line 87, in report
    tree_string_list = self._renderGroup(f, txt, indentation="")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/h5tree.py", line 173, in _renderGroup
    g = self._renderGroup(value, itemname, indentation + "  ", md)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/h5tree.py", line 173, in _renderGroup
    g = self._renderGroup(value, itemname, indentation + "  ", md)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/h5tree.py", line 156, in _renderGroup
    s += self._renderDataset(value, itemname, indentation + "  ")
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/h5tree.py", line 245, in _renderDataset
    value = self._renderArray(dset, indentation + "  ")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/h5tree.py", line 304, in _renderArray
    r = self._renderNdArray(obj, indentation + "  ")
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/h5tree.py", line 337, in _renderNdArray
    r.append(__render(obj, rank, i, indentation + "  "))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/punx/h5tree.py", line 323, in __render
    part = eval("obj[%s]" % indices)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/prjemian/.conda/envs/bluesky_2023_3/lib/python3.11/site-packages/h5py/_hl/dataset.py", line 758, in __getitem__
    return self._fast_reader.read(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "h5py/_selector.pyx", line 376, in h5py._selector.Reader.read
OSError: Can't synchronously read data (can't open directory: /home/prjemian/.conda/envs/bluesky_2023_3/lib/hdf5/plugin)