Currently SQw objects write and store the computed pixel information i.e. qx, qy, qz, E which are $\mathcal{O}\left(n{\text{run}} n{\text{Efix}} n{\text{det}} n{\text{Ebin}} \right) \times 4$ data.
Instead, it is possible to store only the $E{fix}$, detectors, $E{bin}$ arrays and the mapping of the pixels to the requisite indices in these arrays (already stored as det_id, en_id, run_id) to compute qx, qy, qz, E on-the-fly (these expansions are available in e.g. calculate_qw_pixels2). This instead results in $\mathcal{O}\left(n{\text{run}} (n{\text{Efix}} + n{\text{det}} + n{\text{Ebin}} ) \right)$ (given unique detector compression, this may in fact be smaller than this; signal and error and *_id will still need to be stored as $\mathcal{O}\left(n{\text{run}} n{\text{Efix}} n{\text{det}} n{\text{Ebin}} \right)$ )
Not only this, but it should be possible to eliminate zero-counts (possibly following a low-pass filter stage) from the storage by dropping these detector results from the mapping. This could be done by:
splitting the detectors into "signalling"-"non-signalling" and dumping as separate blocks
by deliberately ordering these to the bottom of the data arrays
It would then be possible to drop the signal & error components for these pixels and assume any "empty" detector signal is 0. This will inherently further reduce the volume of stored data.
Initial estimates for standard data suggest that these could result in savings ~5x with respect to current storage.
Currently SQw objects write and store the computed pixel information i.e.
qx, qy, qz, Ewhich are $\mathcal{O}\left(n{\text{run}} n{\text{Efix}} n{\text{det}} n{\text{Ebin}} \right) \times 4$ data.Instead, it is possible to store only the $E{fix}$, detectors, $E{bin}$ arrays and the mapping of the pixels to the requisite indices in these arrays (already stored as
det_id,en_id,run_id) to computeqx, qy, qz, Eon-the-fly (these expansions are available in e.g.calculate_qw_pixels2). This instead results in $\mathcal{O}\left(n{\text{run}} (n{\text{Efix}} + n{\text{det}} + n{\text{Ebin}} ) \right)$ (given unique detector compression, this may in fact be smaller than this;signalanderrorand*_idwill still need to be stored as $\mathcal{O}\left(n{\text{run}} n{\text{Efix}} n{\text{det}} n{\text{Ebin}} \right)$ )Not only this, but it should be possible to eliminate zero-counts (possibly following a low-pass filter stage) from the storage by dropping these detector results from the mapping. This could be done by:
It would then be possible to drop the
signal&errorcomponents for these pixels and assume any "empty" detector signal is0. This will inherently further reduce the volume of stored data.Initial estimates for standard data suggest that these could result in savings ~5x with respect to current storage.