SGpp / DisCoTec

MPI-based code for distributed HPC simulations with the sparse grid combination technique. Docs->(https://discotec.readthedocs.io/)
https://sparsegrids.org/
GNU Lesser General Public License v3.0
8 stars 7 forks source link

Compression w/ LZ4 and MPI-IO #119

Closed freifrauvonbleifrei closed 1 year ago

freifrauvonbleifrei commented 1 year ago

write and read the (sparse grid) data in lz4 format: each rank in the output communicator writes its own data to a frame, as slice in the output file. The first rank, additionaly, writes a header frame containing the file offsets as very first frame. This way, when decompressing each rank will know where to read.