DiamondLightSource / httomo

High-throughput tomography pipeline
https://diamondlightsource.github.io/httomo/
Other
5 stars 4 forks source link

Add flag to specify number of frames per-chunk in intermediate hdf5 data #367

Closed yousefmoazzam closed 2 months ago

yousefmoazzam commented 3 months ago

Built on top of work done in #271

This adds the --frames-per-chunk flag which enables the ability to:

A value of 0 will turn chunking off (ie, write the hdf5 data as contiguous), and values > 0 will be the number of frames per-chunk.

Acceptance criteria checklist:

yousefmoazzam commented 3 months ago

To address the problem of if the value given to the --frames-per-chunk flag exceeds the length of the slicing dim (ie, the chunk shape exceeds the boundaries of the data to be written), when this happens, the frames per-chunk value used is 1 instead of the flag value.

If other behaviour is preferred, feel free to mention, I'll open this for review now.

dkazanc commented 2 months ago

Actually by enabling export OMPI_MCA_fcoll=dynamic fixed the PMIX related issue and I can confirm that I can run with more than one frame per chunk.