There is an ongoing discussion about sub-pixel handling (done upstream), resulting in quite large effective pixel counts in the NeXus files (I have seen numbers from 1024^2 to 5000^2 thrown around). At least towards the higher end, I expect that grouping into pixels becomes increasing inefficient (falling out of CPU cache, ...). We may want to avoid doing this automatically when loading, keeping a flat list of events initially. Of course grouping into pixels later on may still be required, but maybe that can be more manageable with more control of resolution.
To keep the list of pixels flat, we obviously would need to have information such as pixel position for each event. Naively, this would also suffer from CPU cache problems, if the list of positions does not fit into, say L3 cache.
Under the assumption that the positions are regular, we could decompose into 1-D x and y vectors, performing 2 small lookups instead of 1 large. We would need to compute x and y pixel indices from the flat pixel ID (div and mod).
This is just a note for the future to keep in mind, in case we indeed experience performance issues. No need to take any action now.
There is an ongoing discussion about sub-pixel handling (done upstream), resulting in quite large effective pixel counts in the NeXus files (I have seen numbers from 1024^2 to 5000^2 thrown around). At least towards the higher end, I expect that grouping into pixels becomes increasing inefficient (falling out of CPU cache, ...). We may want to avoid doing this automatically when loading, keeping a flat list of events initially. Of course grouping into pixels later on may still be required, but maybe that can be more manageable with more control of resolution.
To keep the list of pixels flat, we obviously would need to have information such as pixel position for each event. Naively, this would also suffer from CPU cache problems, if the list of positions does not fit into, say L3 cache.
Under the assumption that the positions are regular, we could decompose into 1-D x and y vectors, performing 2 small lookups instead of 1 large. We would need to compute x and y pixel indices from the flat pixel ID (div and mod).
This is just a note for the future to keep in mind, in case we indeed experience performance issues. No need to take any action now.