The current design of cli/simulate_pixels.py saves all output at the end of its run. For larger inputs, this tends to result in out-of-memory issues since everything doesn't fit in memory at once. Since h5 datasets support extending in-place, we should be writing to the file continuously (store after every event?), which would eliminate this problem.
soleti
commented
2 years ago
The current design of
cli/simulate_pixels.pysaves all output at the end of its run. For larger inputs, this tends to result in out-of-memory issues since everything doesn't fit in memory at once. Since h5 datasets support extending in-place, we should be writing to the file continuously (store after every event?), which would eliminate this problem.