search

McStasMcXtrace / McCode

The home of the McStas (neutrons) and McXtrace (x-rays) Monte-Carlo ray-tracing instrument simulation codes.
https://github.com/McStasMcXtrace/McCode/wiki
GNU General Public License v3.0
77 stars 54 forks source link

Storing event data in NeXuS efficiently and compressed #1050

Open aglavic opened 3 years ago

aglavic commented 3 years ago

I'm using the Monitor_nD list mode for storing neutron events regularly as it allows more sophisticated analysis after the simulation. This becomes an issue when better statistics is required or two detectors with different flux are run together as the file size becomes extremely large. (>100GB)

While NeXuS/HDF5 supports compression very efficiently, McStas uses uncompressed files. In addition, compression of the file directly does not help, as the chunk size for events is always (1xN). I'm using h5repack to change the chunck size to e.g. 3072xN and compress the events with very good results: h5repack -f /entry1/data/tof_detector_list_p_x_y_t_L/events:GZIP=5 \ -l /entry1/data/tof_detector_list_p_x_y_t_L/events:CHUNK=3072x5 \ fin fout

It would be very convenient to include these options in the McStas file writer directly to safe the extra effort and time.

willend commented 3 years ago

Dear Artur,

Thanks for reporting / suggesting this improvement.

For several reasons we are already thinking of how we should eventually improve the NeXus implementation in McStas, one of which being that the so-called "nexus C API" / napi.h approach we use is being deprecated by the NeXus team.

We will keep your suggestion in mind when we find time and resources to do something in this direction...