search

ScottishCovidResponse / SCRCIssueTracking

Central issue tracking repository for all repos in the consortium
6 stars 0 forks source link

HDF5 outputs with MPI #348

Open claireh93 opened 4 years ago

claireh93 commented 4 years ago

Following on from #309, once we have an MPI set up we should think about writing data directly from each node, so that the full dataset doesn't need to be gathered. Blocked until #4 goes through.

github-actions[bot] commented 4 years ago

Heads up @claireh93 @richardreeve @jwscook @sdl1 @wytbella - the "Simulation.jl" label was applied to this issue.

jwscook commented 4 years ago

On zulip there is chat about using https://github.com/JuliaGeo/NetCDF.jl instead. We need to be sure that NetCDF offers a method of parallel reading + writing like HDF5 does.

jwscook commented 4 years ago

https://github.com/JuliaIO/HDF5.jl/blob/master/test/mpio.jl

richardreeve commented 4 years ago

Oh, that's cool. It's looking like nobody has the energy to use netCDF for this first implementation, so we're safe for now.

I was wondering given how much better some aspects of the HDF5.jl code is, whether if we did switch we could read files using NetCDF.jl, and write the output format too, but then close the file, and reopen it using HDF5.jl and write the output slices using that package?