Currently, the tomogram data is converted from float32 to uint16 spread over two hdf5 files (most- vs least-significant byte)
hdf5-byte/msb/{sample}.h5
hdf5-byte/lsb/{sample}.h5
These hdf5-files contain all 4-6 subvolumes in full, which means that when they are read, we get duplicted overlaps, with discontinuities and unphysical sub-images as result. The volume matching is designed to combine subvolume images smoothly, matching the overlaps. The volume-matching shifts are stored in the MSB hdf5-file, but not currently used.
Proposed solution:
Due to the extreme slowness of h5py, and the unwieldiness of C and C++ implementations of hdf5, we now dump to a simple binary blob before processing. Currently, this simply reproduces the problematic subvolume overlaps. I propose that we keep the msb/lsb hdf5 files as "archives" that include all the original information, and instad implement in scripts/h5tobin.py that it does the volume matched subvolume recombination (reading the shifts from msb/{sample}.h5) as it produces the blob.
Currently, the tomogram data is converted from float32 to uint16 spread over two hdf5 files (most- vs least-significant byte) hdf5-byte/msb/{sample}.h5 hdf5-byte/lsb/{sample}.h5 These hdf5-files contain all 4-6 subvolumes in full, which means that when they are read, we get duplicted overlaps, with discontinuities and unphysical sub-images as result. The volume matching is designed to combine subvolume images smoothly, matching the overlaps. The volume-matching shifts are stored in the MSB hdf5-file, but not currently used.
Proposed solution:
Due to the extreme slowness of h5py, and the unwieldiness of C and C++ implementations of hdf5, we now dump to a simple binary blob before processing. Currently, this simply reproduces the problematic subvolume overlaps. I propose that we keep the msb/lsb hdf5 files as "archives" that include all the original information, and instad implement in scripts/h5tobin.py that it does the volume matched subvolume recombination (reading the shifts from msb/{sample}.h5) as it produces the blob.