search

jacg / petalo

PET scanner image reconstruction
1 stars 4 forks source link

Parallelize reading LORs #23

Open jacg opened 2 years ago

jacg commented 2 years ago

This is the item which was not addressed in the work that closed #18.

After #18, loading the LORs from file takes about 25% of the total reconstruction time on frontend1petalo with unlimited core use. Using 25 threads, it takes around 15% of the total time.

Seems unlikely that we'll be able to do much about it, unless the hdf5 crate provides something that helps.

Overall, I suspect that further optimizations with the current design are probably not worth the effort. More significant gains will probably come from switching to data-oriented design (#19) and Apache Arrow/Parquet.

jacg commented 2 years ago

Currently we take thousands of files produced by MC jobs and reconstruct LORs into a single, enormous LORs file. Reading LORs from this single file is the bottleneck.

Reconstructing LORs into fewer, separate files, would probably make the task of parallelizing LOR reading very easy.

jacg commented 1 year ago

The Rust HDF5 crate uses a version of the underlying C library that contains a global lock, so we're limited to reading only one HDF5 file per process. Parallel IO in HDF5 seems to require OpenMP. Yet another reason to replace HDF5 with some 21st century technology such as Arrow/Parquet.