search

kif / multianalyzer

Rebinning of powder diffraction data taken with multiple analyzer crystals and a large area detector
MIT License
0 stars 2 forks source link

Scan file is locked by multianalyzer #11

Closed woutdenolf closed 2 years ago

woutdenolf commented 2 years ago

To be used online, this code should never lock the file and should retry in case of failure (concurrent read/write).

I could do one of these things:

  1. use the h5py utils from silx in multianalyzer
  2. or do the hdf5 reading outside and multianalyzer receives the data in memory

@kif What do you prefer?

kif commented 2 years ago

The usage of silx.io is the easiest for now. Did you have a look at the profiling results? Most of the time is spent in reading and writing HDF5 files. So memory access would be desirable but the amount of data to share (several GB) makes it impracticable when processing on a different computer. Memcached does not scale well above dozens of MB. So probably going via HDF5 is still the best.