search

NorESMhub / noresm2cmor

A command line tool for cmorizing NorESM output
http://noresmhub.github.io/noresm2cmor/
5 stars 16 forks source link

Add CMIP6_6hrLev/6hrPlev/6hrPlevPt tables #109

Closed YanchunHe closed 7 months ago

YanchunHe commented 4 years ago

Fields in 6hr-averaged output (cam.h2)

The Following fields are now supported:

                 'bldep       ','PBLH          ','          ',
                 'hurs        ','RHREFHT       ','percent   ',
                 'hus4        ','Q             ','          ',
                 'pr          ','PRECT         ','kg m-2 s-1',
                 'psl         ','PSL           ','          ',
                 'sfcWind     ','U10           ','          ',
                 'tas         ','TREFHT        ','          ',
                 'wap4        ','OMEGA         ','          ',
                 'zg1000      ','Z1000         ','          ',

The Following fields are not available in the model output:

!                'bs550aer    ','BS550AER/DAYFOC','DAYFOC   ',
!                'uas         ','UAS           ','          ',
!                'vas         ','VAS           ','          ',

Fields in 6hr-instantaneous output (cam.h3)

The Following fields are now supported:

                 'hus         ','Q             ','          ',
                 'ps          ','PS            ','          ',
                 'ta          ','T             ','          ',
                 'ua          ','U             ','          ',
                 'va          ','V             ','          ',
                 'hus27       ','Q             ','          ',
                 'huss        ','QREFHT        ','          ',
                 'psl         ','PSL           ','          ',
                 'sfcWind     ','U10           ','          ',
                 'ta          ','T             ','          ',
                 'tas         ','TREFHT        ','          ',
                 'ts          ','TS            ','          ',
                 'ua          ','U             ','          ',
                 'va          ','V             ','          ',
                 'zg27        ','Z3            ','          ',
                 'zg500       ','Z500          ','          ',

The Following fields are not available in the model output: (6hr output from land model is totally missing) (mrsol, mrsos,snw should come from clm output)

!                'mrsol       ','SOILLIQ+SOILICE','         ',
!                'mrsos       ','SOILWATER_10CM','          ',
!                'snw         ','SNOWICE+SNOWLIQ','         ',
!                'tsl         ','TSL           ','          ',

Sample output

Commit

UPDATED: 30 OCT 2019, correct 6 hourly average and instantaneous file streams. commit: 210389a, 8df4e53

OskarMETNO commented 4 years ago

Is it really correct that the instantaneous fields (6hrLev) have timestamps at 03h, 09h, 15h and 21h? Shouldn't it be 00/06/12/18? Or has this changed from CMIP5 to CMIP6? (Edit: Looking at ESGF, it seems most models have 00/06/12/18, but some have 03/09/15/21.)

YanchunHe commented 4 years ago

First of all, the 6hourly instantaneous fields are those has tag '6hrPlevPt" instead of 6hrLev (which is 6hourly average).

I checked the LM historical run, as an example, the model output should indeed has time tag at 00/06/12/18: ncdump -v time -t NHIST_f19_tn14_20190710.cam.h3.1959-10-20-21600.nc ... "1959-12-29 06", "1959-12-29 12", "1959-12-29 18", "1959-12-30", "1959-12-30 06", "1959-12-30 12", "1959-12-30 18", "1959-12-31", "1959-12-31 06", "1959-12-31 12", "1959-12-31 18", "1960-01-01" ;

While in the cmorized data: ncdump -t -v time hus_6hrPlevPt_NorESM2-LM_historical_r1i1p1f1_gn_195001010300-195912312100.nc "1959-12-30 03", "1959-12-30 09", "1959-12-30 15", "1959-12-30 21", "1959-12-31 03", "1959-12-31 09", "1959-12-31 15", "1959-12-31 21" ; They are tagged at 03/09/15 and 21.

The time information is read from the model output and the output file tag is determined by the CMOR library. I have not touched this part of the cmorization tool, so I have now no clear idea about the mechanism in cmor library.

If the noresm2cmor tool has done something before passing to the cmor library?

@IngoBethke , maybe you have some impression and can recall on this?

btw, I see some other modelling groups use the tool 'nctime' to quality-check the time-axis of the cmorized data (https://github.com/Prodiguer/nctime).

@monsieuralok , is it possible to install and run this tool for the cmorized NorESM2 data too? Look like a simple tool.

OskarMETNO commented 4 years ago

I thought 6hrLev is supposed to be instantaneous. In the metadata it says ta:cell_methods = "area: mean time: point" At least for CORDEX-MIP (where a lot of these variables are requested) it would make more sense to have instantaneous values to be used for dynamical downscaling.

Edit: Ingo beat me to it in an e-mail. His reply is included here as well for reference:

For CMIP5, I think to remember that all 6-hourly data used to be instantaneous.

It looks like this has somewhat changed in CMIP6. The fields in e.g. 6hrPlev are supposed to be time-means (see cell_methods in https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_6hrPlev.json ) whereas the fields in 6hrLev are supposed to be instantaneous (see https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_6hrLev.json ).

YanchunHe commented 4 years ago

Sorry for the late response!

I have a typo above. Yes, the 6hrLev should be 6-hr instantaneous on altitude heights, and 6hrPlev is 6-hr average on Pressure levels, and 6hrPlevPt is instantaneous on Plevels.

Actually, there is a 6-hourly average variable, bs550aer,on levels (6hrLev) in CMIP6, but it is not available from NorESM2 output, so there is not actually cmorized. correction: I find in the https://github.com/PCMDI/cmip6-cmor-tables/blob/master/Tables/CMIP6_6hrLev.json, this is also instantaneous, but in the CMIP6 data request Google sheet, it is on 6-hourly average (line number 25). So the google sheet is wrong or not update-to-date to the CMIP6_6hrLev.json

YanchunHe commented 4 years ago

I see that the time stamps is shifted from 0/6/12/18 to 3/9/15/21. This is very likely done in our noresm2cmor implementation, that the time values is used as the average of the first time slice and the last time slice. Something like:

tval=0.5*(tbnds(1,1)+tbnds(2,1))

I may fix this in the future release.

Is this very important to republish new data version of published experiment? or we can just create an ESGF known-issue?

YanchunHe commented 4 years ago

This has been fixed in commit: 9ef437e91df26aab0f52c7a8e32d81f484c59975

The 3hrly and 6hourly data of NorESM2-LM historical will be reprocessed with data version v20200218, and be published to ESGF later.

Will not reprocess published data.

Now the timestamp for 6hr instantaneous is like: "1950-01-29 06", "1950-01-29 12", "1950-01-29 18", "1950-01-30", "1950-01-30 06", "1950-01-30 12", "1950-01-30 18", "1950-01-31", "1950-01-31 06", "1950-01-31 12", "1950-01-31 18", "1950-02-01" ; and for 3hr instantaneous data is like: "1950-01-30 15", "1950-01-30 18", "1950-01-30 21", "1950-01-31", "1950-01-31 03", "1950-01-31 06", "1950-01-31 09", "1950-01-31 12", "1950-01-31 15", "1950-01-31 18", "1950-01-31 21", "1950-02-01" ;

YanchunHe commented 4 years ago

data will be found under: /projects/NS9034K/CMIP6/.cmorout/NorESM2-LM/historical/v20200218

YanchunHe commented 4 years ago

fixed in commit: 9ef437e

Data of above referred experiments were cmorized with correct time stamp, and are published to ESGF.

YanchunHe commented 3 years ago

by Ø.Seland: (issue #307)

Fields in 6hr-instantaneous output (cam.h3) in CMIP6_6hrPlevPt

'snowmxrat ','QSNOW

oyvindseland commented 2 years ago

The error in timestamp described above in NorESM2-LM does also exist for some fields in the historical ensemble member 1 in NorESM2-MM see copied e-mail to noresm-ncc from Trude Eidhammer. Can this be reprocessed or is it just described as a known issue in the ESGF information @YanchunHe @monsieuralok I have confirmed that this is an issue, but only seem to exist in ensemble member 1.

"I am using the NorESM2-MM CMIP6 runs to create forcing to downscale CMIP6 models. Most of the atmospheric files are output at 0, 6, 12 and 18 UTC. However, the historic temperature data is at 3, 9, 15, and 21 UTC (see for example this type of file ta_6hrLev_NorESM2-MM_historical_r1i1p1f1_gn_195001010300-195912312100)

I downloaded the files from ESGF. Do you still have 6 hourly historical data at timestamps 0,6,12, and 18?"

maosier commented 2 years ago

The files in the NorESM2-MM historical runs almost start at 03, 09, 15, 21 UTC, while the same output in the NorESM2-MM ssp runs start at 00, 06, 12, 18 UTC. I'm wondering the historical run is only a timestamp error and the data is actually correct, or the timestamp and data are both at the 03, 09, 15, 21 UTC?

yfa008 commented 1 year ago

After reading the discussions above, I am surprised that the surface air temperature and humidity variables (tas, huss etc.) published on ESGF (huss_6hrPlevPt, tas_6hrPlevPt, tas_6hrPlev, ps_6hrLev etc.) were labeled as having been interpolated to pressure levels. They should correspond to the original variables TREFHT and QREFHT in the .cam.h5.* history files, which were calculated by the model with reference to 2-m height above the ground. The original data of TREFHT and QREFHT should be directly renamed to tas and huss according to CMIP6 definition of these surface climate variables. Why did they need to be interpolated to pressure levels? Or are 6hrPlevPt and 6hrPlev just mis-labels?

YanchunHe commented 1 year ago

After reading the discussions above, I am surprised that the surface air temperature and humidity variables (tas, huss etc.) published on ESGF (huss_6hrPlevPt, tas_6hrPlevPt, tas_6hrPlev, ps_6hrLev etc.) were labeled as having been interpolated to pressure levels. They should correspond to the original variables TREFHT and QREFHT in the .cam.h5.* history files, which were calculated by the model with reference to 2-m height above the ground. The original data of TREFHT and QREFHT should be directly renamed to tas and huss according to CMIP6 definition of these surface climate variables. Why did they need to be interpolated to pressure levels? Or are 6hrPlevPt and 6hrPlev just mis-labels?

This issue is discussion the timestamp. You may refer to issue #316, which is about interpolation of data from sigma2 coordinate to selected pressure levels.

Although the surface variables tas, hurs (not huss. huss is for 3hr output), etc are defined in the 6hrPlev, they are not of course interpolated to different pressure levels, but defined as a single 'height2m' level. And they are cmorized and should be available on ESGF:

$ search project=CMIP6 activity_id=CMIP institution_id=NCC source_id=NorESM2-LM experiment_id=historical variant_label=r1i1p1f1 table_id=6hrPlev

new  CMIP6.CMIP.NCC.NorESM2-LM.historical.r1i1p1f1.6hrPlev.hus.gn.v20191108
new  CMIP6.CMIP.NCC.NorESM2-LM.historical.r1i1p1f1.6hrPlev.pr.gn.v20191108
new  CMIP6.CMIP.NCC.NorESM2-LM.historical.r1i1p1f1.6hrPlev.psl.gn.v20191108
new  CMIP6.CMIP.NCC.NorESM2-LM.historical.r1i1p1f1.6hrPlev.tas.gn.v20191108
new  CMIP6.CMIP.NCC.NorESM2-LM.historical.r1i1p1f1.6hrPlev.zg1000.gn.v20191108
new  CMIP6.CMIP.NCC.NorESM2-LM.historical.r1i1p1f1.6hrPlev.hurs.gn.v20191108

So those surface variables are available.

btw, the 6 hourly data are in the output of 'cam.h2', and 'cam.h3'. And the 3 hourly data are in the 'cam.h4' and 'cam.h5' stream.

For missing tas and huss, etc for 3 hourly output, they are included in the cmorization variable list. I can not recall why they were not cmorized, and the log files for cmorizing these 3 hourly data were missing during some disk/installation change phases (the log files are not so important so that they are not committed to github).

Anyway, it is another issue, and I would refer to #41 to keep track on this. The NS9034K project storing the NorESM cmorized output is closed recently due to NIRD migration, I will look at this later.

YanchunHe commented 1 year ago

tas, hurs (correpsonds to TREFHT and QREFHT) are not available in *.h4. (3hourly average) files, so they can not be cmorized.

I will close this issue if no other comments.

yfa008 commented 1 year ago

Hi Yanchun, thanks for looking into this and your reply. how about 6hourly tas, hurs or huss. Are they available? If so, it would be a valuable addition to the impact research community who look at sub-daily patterns.

Yuanchao

Yuanchao Fan Assistant Professor Institute of Environment and Ecology Tsinghua Shenzhen International Graduate School

Tsinghua University

Shenzhen 518055, China email: @.*** tel: +86 18026984060

On Wed, Jun 7, 2023 at 8:05 PM Yanchun He @.***> wrote:

tas, hurs (correpsonds to TREFHT and QREFHT) are not available in *.h4. (3hourly average) files, so they can not be cmorized.

I will close this issue if no other comments.

— Reply to this email directly, view it on GitHub https://github.com/NorESMhub/noresm2cmor/issues/109#issuecomment-1580671581, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGGNJV2Z36HLQHEOCT5VGG3XKBU7DANCNFSM4JGPMX2A . You are receiving this because you commented.Message ID: @.***>

YanchunHe commented 1 year ago

Yes, 6-hourly tas, hurs are already cmorized and available under the 6hrPlev table_id.

YanchunHe commented 8 months ago

Quote "O. Landgren"

"...there is likely a similar issue for some of the 3-hourly fields for NorESM2-MM:

For example tslsi is on 01:30:00, 04:30:00, 07:30:00... instead of 0, 3, 6... NorESM2-MM/historical/r1i1p1f1/3hr/tslsi/gn/v20191108/tslsi_3hr_NorESM2-MM_historical_r1i1p1f1_gn_201001010130-201412312230.nc

Also related to issue #41

YanchunHe commented 7 months ago

@monsieuralok

Could you please report an Erratum for all the '6hrLev' and '6hrPlevPt', datasets earlier than v202018, that the time stamp should be shift 3 hours to more recent. E.g, shift 03:00/09:00/15:00/21:00 to 06:00/12:00/18:00/24:00.

For the 3 hourly data, you can report that for all the frequency of '3hrPt' , the time stamp should also be increased by 1.5 hours. The frequency '3hr' is not affected.

YanchunHe commented 7 months ago

@monsieuralok

Could you please report an Erratum for all the '6hrLev' and '6hrPlevPt', datasets earlier than v202018, that the time stamp should be shift 3 hours to more recent. E.g, shift 03:00/09:00/15:00/21:00 to 06:00/12:00/18:00/24:00.

For the 3 hourly data, you can report that for all the frequency of '3hrPt' , the time stamp should also be increased by 1.5 hours. The frequency '3hr' is not affected.

OK, there is summary and correction to 3hourly data:

For the 6-hourly data:

  1. for 6hrLev and 6hrPlevPt, of data version earlier than v20200218, the time should be shifted by 3 hours, e.g., 03:00->06:00. This should be reported in the Erratum.
  2. The '6hrPlev' dataset has been always, before or after v20200218, using the averaged time for each averaging period. This is intended. But this can also be reported, not as error, but for an alert. And this may be changed in the future without any post processing of the time but use the same as the model (e.g., for CMIP7).

For 3 hourly data (related to issue #41)

  1. For the CF3hr and E3hrPt datasets, versions before v20200218 should be shifted by 1.5 hours, e.g., 01:30->03:00. This should be reported in the Erratum.
  2. for E3hr, averaged time is used, and no correction is needed. But this can also reported, as the same as '6hrPlev'
  3. for 3hr datasets, the fields should be treated separately for individual variables. More specifically:
    • Fields at certain time points within the 3 hourly sampling are affected in all versions, both before and after v2020218. They should be shifted by 1.5 hours. This should be reported in the Erratum. Including:
    • huss
    • mrsos
    • ps
    • tas
    • tslsi
    • other fields on 3 hourly time average are not affected. but they can reported as an alert.
    • mrro
    • pr
    • prc
    • prsn

One may use CDO to fix the time stamp, for example: cdo shifttime,+3hour infile outfile for shifting the affected the 6 hourly data.

OskarMETNO commented 7 months ago

Just a small typo I assume in the last post. The 3-hourly data should be shifted by 1.5 hours (and 6-hourly data should be shifted by 3 hours). And in an errata post it would also be good to be explicitly clear that the times should be shifted by adding 1.5 and 3 hours, respectively, and not by subtracting it.

YanchunHe commented 7 months ago

Just a small typo I assume in the last post (in the above). The 3-hourly data should be shifted by 1.5 hours (and 6-hourly data should be shifted by 3 hours). And in an errata post it would also be good to be explicitly clear that the times should be shifted by adding 1.5 and 3 hours, respectively, and not by subtracting it.

Thanks, Oskar. I corrected the typo. And for the 3 hourly data of time point output, they should be shifted by 1.5 hours for all versions, not only before v20200218 (I corrected this too in the above).

monsieuralok commented 7 months ago

@YanchunHe does it includes also datasets version v20200218 for 6hrLev and 6hrPlevPt ? or only all version before v20200218 ? Also, I donot see any datasets with E3hrPt on ESGF node.

YanchunHe commented 7 months ago

@YanchunHe does it includes also datasets version v20200218 for 6hrLev and 6hrPlevPt ? or only all version before v20200218 ? Also, I donot see any datasets with E3hrPt on ESGF node.

before v20200218.

Then there is ability to cmorize the E3hrPt variable, but there are not available from model output. That is OK.

monsieuralok commented 7 months ago

@YanchunHe https://errata.ipsl.fr/static/view.html?uid=97352b25-4e4b-4a1a-a5a0-d28c32c0cc87 https://errata.ipsl.fr/static/view.html?uid=5fb6c827-3ecb-3b98-8c71-9beaae70a847 https://errata.ipsl.fr/static/view.html?uid=3f489161-0c87-84e8-9f73-f23fac5021ae Opened for 6hrLev, 6hrPlevPt CF3hr E3hrPt 3hr You can check it and will open alert for 3hr 'E3hr and 6hrPlev'

YanchunHe commented 7 months ago

@YanchunHe https://errata.ipsl.fr/static/view.html?uid=97352b25-4e4b-4a1a-a5a0-d28c32c0cc87 https://errata.ipsl.fr/static/view.html?uid=5fb6c827-3ecb-3b98-8c71-9beaae70a847 Opened for 6hrLev, 6hrPlevPt CF3hr and E3hrPt You can check it and will open for 3hr and alert for 'E3hr and 6hrPlev'

Looks good. Thanks!

monsieuralok commented 7 months ago

@YanchunHe https://errata.ipsl.fr/static/view.html?uid=0193293b-e2ef-3935-bb6f-b8ec9f5f5603 I have create issue for Alert you can check.

YanchunHe commented 7 months ago

@YanchunHe https://errata.ipsl.fr/static/view.html?uid=0193293b-e2ef-3935-bb6f-b8ec9f5f5603 I have create issue for Alert you can check.

Looks good to me. Thanks!