Improve postprocessing - Githubissues

ISI-MIP / attrici

Produce counterfactual climates for ISIMIP.

GNU General Public License v3.0

9 stars 2 forks source link

Improve postprocessing #71

Closed matthiasmengel closed 4 years ago

matthiasmengel commented 4 years ago

Until now, we only worked with subsets of the data, maximum 5. The full dataset will be 25x as large. We need to think how to efficiently postprocess the timeseries into a netcdf file. HDF5, xarray or parallel netcdf are to explore. merge_parallel.py and merge_submit.sh may be reused.

matthiasmengel commented 4 years ago

Takes 100 minutes to write the .h5 timeseries to one large netcdf. This is still fine. So this issue has low priority.

matthiasmengel commented 4 years ago

To have our output traceable, let us bring githash and the runid as a global attribute to the netcdf file. Further:

rename variables to its standard name (not y but tas).
if we want to save the original data, let us name it tas_orig or similar
allow to pick only specific variables from the .h5 timeseries files

matthiasmengel commented 4 years ago

Attributes now added, see c4e469e5f754442. Closing this.