search

DedalusProject / dedalus

A flexible framework for solving PDEs with modern spectral methods.
http://dedalus-project.org/
GNU General Public License v3.0
492 stars 115 forks source link

Fixes virtual file merge logic & removes deprecated distributed set merge logic #206

Open evanhanders opened 2 years ago

evanhanders commented 2 years ago

This PR addresses #205 and updates post.py to reflect Dedalus' new output paradigm by removing deprecated functions for merging partial files. Functionality for merging partial files => complete files after a simulation is updated. Tests have been added for each of the new parallel output types ('virtual', 'mpio', 'gather').

All output and parallel output tests now pass. There are some failing ncc tests, but those should be unaffected by these commits.

kburns commented 2 years ago

It should be possible to simply merge virtual datasets into concrete datasets within the existing set file using the merge_task method on the new H5VirtualFileHandler class. I think this should remove the need for most of the logic in post.py, including creating a new file, copying scales, etc.

evanhanders commented 2 years ago

Hi Keaton, I didn't know about that method! I'll take a look and try to get that implemented. I've marked the PR as a draft for now.