search

SWIFTSIM / swiftsimio

Python library for reading SWIFT data. Uses unyt and h5py.
GNU Lesser General Public License v3.0
16 stars 13 forks source link

Concatenate multi-file snapshots? #110

Open kyleaoman opened 2 years ago

kyleaoman commented 2 years ago

Would it make sense to support multi-file snapshots (snap.0.hdf5; snap.1.hdf5; ...) so that they can be accessed by a single SWIFTDataset that simply concatenates the arrays from each part? In other words, the user shouldn't need to know that there's more than one file. I have a bit of code that I used on EAGLE-like snapshots. I don't think I'd suggest using it as-is, but could serve as inspiration? https://github.com/kyleaoman/simfiles/blob/master/simfiles/_hdf5_io.py

MatthieuSchaller commented 2 years ago

SWIFT now outputs a meta-snapshot that can be used to read the distributed snapshots. SWIFTsimio can make use of that.

(Unfortunately sibelius was run before this feature was added)

However, that is still not going to be super efficient if you want to read the whole file. There is no parallel-reading feature. But if you just want a single region then it works very well.