EC-Earth / ece2cmor3

Post-processing and cmorization of ec-earth output
Apache License 2.0
13 stars 6 forks source link

ln(sp) vs sp in GG/SH files #21

Closed pabretonniere closed 7 years ago

pabretonniere commented 7 years ago

Hi @goord ,

CMORizing our files from the latest EC-Earth version, it fails because it is looking for the surface pressure in the SH files to calculate the levels. It is due to the commit 3842 (https://dev.ec-earth.org/projects/ecearth3/repository/revisions/3842) that you made in the primavera branch saying to output the pressure level in the GG files and not in the SH as before (keeping only lnsp in the SH). I can update the code to tell ece2cmor to compute the levels from exp(ln(sp)) but I wanted to check if you hadn't already had the same problem. I think we would lose less information than using ps from the GG files.

Thanks, Cheers,

goord commented 7 years ago

Hi Pierre-Antoine,

I noticed this too when I was doing the cmorization at ecmwf. I can commit my changes and then ece2cmor3 will just query the GG file for the surface pressure. Perhaps the best solution would be to add an option to ece2cmor3 to tell it where to look for ps, because it may be in either file...

etiennesky commented 7 years ago

Hi @goord I am not sure that using sp from the GG file since the grid is not the same as the one in the SH file. I think that ece2cmor3 should compute sp on the fly from exp(lnsp) if sp is not found, without any option.

Etienne

etiennesky commented 7 years ago

So according to @pabretonniere sp must be present in the SH files before sending them to cmor library, which would require a pre-processing step with grib_filter.

Would it not be simpler then to simply re-add sp to the IFS output in the SS files ?

Also is lnsp used at all, in which case we could remove it in favor of sp?

klauswyser commented 7 years ago

Indeed, sp in the GG file and sp in the ICMSH file are not the same beast, but honestly I cannot tell which one is correct and which one isn’t. There are a few fundamental differences between the two:

Clearly, the two surface pressures are not the same, and I would say it depends on the application to say which one is more appropriate. For working with upper air fields I probably woud take lnsp from ICMSH, but for working with SFC data I'd rather had the gridpoint value from ICMGG. Which one should we save for CMIP6?

BTW: whatever ps we decide to save should also have an implication on which orography we are delivering to CMIP6, SFC pressure and orography should correspond to each other.

etiennesky commented 7 years ago

I think that the cmor library requires sp in the SH files, so this question is about where do we get sp from, the GG files (I think better not), exp(lnsp), or add them to the SH files from IFS.

in the openIFS doc on IFS output (https://software.ecmwf.int/wiki/display/OIFS/How+to+control+OpenIFS+output), there is a hint that sp, lnsp and should be enabled, which are not at the moment. To output on pressure, PV or theta level output you must also enable output of the 2D spectral orography, surface pressure & log surface pressure.

klauswyser commented 7 years ago

Can you save sp (134) in ICMSH? I thought you can only save lnsp (151) there, but I may be wrong.

etiennesky commented 7 years ago

And then there if the quite relevant issue which Klaus also raised, do we get sp from SH of GG files for final output ? What was done in cmip5/specs ?

etiennesky commented 7 years ago

Yes Klaus saving sp(134) is possible but it was removed by Gijs some time ago in r3842 "Moved surface pressure (134) from MFP2DF to gridpoint variables".

Also Z(129) was removed in r3844, "Removed 129 from 2d spectral variables, already produced on gridpoints".

I am not sure I understand why these were removed, perhaps this did not do what was intended.

goord commented 7 years ago

If you enable 134 in the MFP2DF namelist, it will appear in the spectral file, if you add it to the MFPPHY namelist it appears in the gridpoint file. Note there are several use cases: 3D fields need hybrid model level information to be stored within the same netcdf file, for this CMOR needs sp, some variables need to be calculated during postprocessing and need the surface pressure for this, and finally we need to report sp in various tables.

goord commented 7 years ago

Etienne, in https://software.ecmwf.int/wiki/display/OIFS/How+to+control+OpenIFS+output they also mention you can output them on gridpts, I thought it didn't matter since we are converting to gridpoints anyway...

klauswyser commented 7 years ago

it didn't matter since we are converting to gridpoints anyway...

It does because they are not equal in gridpoint space.

@etiennesky: spectral orography would be good to have, but only in ICMSH+000000 (similar as gridpoint orography is saved only in ICMGG+000000)

goord commented 7 years ago

Ok, so can I summarize this as follows:

1) For the reconstruction of the hybrid model levels, we should exponentiate lnsp (or use sp) in the spectral representation, this will give the best result along the entire vertical range. 2) For the computation of certain surface fields, we can use sp in the gridpoint file (this is actually necessary in ece2cmor3, it cannot combine variables from the two files) 3) For the mean surface pressure, requested in some of the tables, the spectral version is a better match with the orography

pabretonniere commented 7 years ago

OK, so, in the end, we agree that the easiest (and less memory intensive) option would be to output directly from the model sp both in the GG and SH files, right?

etiennesky commented 7 years ago

@pabretonniere and I think that it would be much more efficient to save sp in the SH files, instead of computing it at the beginning of ece2cmor3 which will further add to the processing time at a small cost of space. This way it is available for other computations by cmor library, as well as for the output consistent with spectral orography.

Do you guys agree?

I can add spectral orography to the ICMSH*+000000 and sp to the 3h and 6h files in the primavera branch and have @pabretonniere test again.

goord commented 7 years ago

Yes that is a good idea, it would also be useful to test the differences between the spectral and gridpoint surface pressure fields after interpolation upon the regular grid. Note that we have to keep ps also in the GG file to calculate the surface specific humidity.

I can make ece2cmor in such a way that it looks for sp in the SH file, and if not found, looks for lnsp and exponentiates it. This is not too much overhead since it is just a surface field, the 3d fields dominate the runtime anyway.

etiennesky commented 7 years ago

OK, I will add sp to the SH files and make sure that the orography is present in both SH and GG initial state files (just the first timestep) .

However, do we still need lnsp in both SH and GG files?

goord commented 7 years ago

No, if sp is in there we don't need lnsp, and I think the diagnostics will continue to work if we remove it.

etiennesky commented 7 years ago

Hi @klauswyser is there any scientific or technical reason why we should save lnsp? If not I will drop it from the output list in favor of sp. Thanks!

klauswyser commented 7 years ago

But honestly, I don't think any of these reasons is strong enough to warrant that we save both sp and lnsp in both the ICMGG and the ICMSH file, for simplicity I would just save sp.

etiennesky commented 7 years ago

I just committed the proposed changes to the primavera branch in r4236 . @pabretonniere needs to test and fix the python grib filter with the files I have after this change.