search

NorESMhub / noresm2cmor

A command line tool for cmorizing NorESM output
http://noresmhub.github.io/noresm2cmor/
5 stars 16 forks source link

[CMIP6 CMORization & ESGF-publication] CMOR-ization of first two PAMIP experiments #28

Open YanchunHe opened 5 years ago

YanchunHe commented 5 years ago

The first two PAMIP experiments pdSST-pdSIC and pdSST-futArcSIC are finished and ready to be CMOR-ized. Both experiments consist of 100 members. I have not been able to run a single experiment with all 100 members because it is very CPU demanding, so each experiment consists of four cases with 25 members in each case.

PAMIP experiment pdSST-pdSIC: Members 1-25: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem1-25_f19_mg17_20190819 Members 26-50: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem26-50_f19_mg17_20190819 Members 51-75: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem51-75_f19_mg17_20190819 Members 76-100: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem75-100_f19_mg17_20190819 PAMIP experimet pdSST-futArcSIC: Members 1-25: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem1-25_f19_mg17_20190819 Members 26-50: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem26-50_f19_mg17_20190819 Members 51-75: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem51-75_f19_mg17_20190819 Members 76-100: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem76-100_f19_mg17_20190819 Note that the model forces the ensemble member numbering in the file names to be 0001 - 0025 in all cases, so for instance member 1 in case NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem26-50_f19_mg17_20190819 is really member 26 etc. So unfortunately, the information on which ensemble members resides in which case can only be found in the case name itself (note the substrings "mem1-25", mem26-50" etc). Let me know if this is not well explained.

The experiments are 14 months long starting from 1. April 2000 and ending 31. May 2001. The first two months should be considered spin-up and must not be included in the CMOR-ized files. This means that the first day in the CMOR-ized files should be 1. June 2000 and the last day should be 31. May 2001 (a full year).

I have gotten a few requests from people who are writing up papers on the PAMIP data, and specific variables they are interested in include monthly-mean temperature, zonal wind, sea level pressure, and geopotential height, is if possible it would be nice if these are included in the initial CMOR-ized output. I have also had requests for the Eliassen-Palm fluxes, but I guess we still have some work to do to get those.

(L. Graff)

YanchunHe commented 5 years ago

pdSST-pdSIC and pdSST-futArcSIC are cmorized and stored now under: /tos-project1/NS9034K/CMIP6/.cmorout/NorESM2-LM

But does not pass PrePARE quality-check, with "list index out of range" error. see Issue #31

YanchunHe commented 5 years ago

Qality-check PrePARE passed, see Issue #31 CMORized data is ready to publish to ESGF. See Issue #37

IngoBethke commented 5 years ago

Member 94 of pdSST-futArcSIC contains fewer files than the other members according to the checksum file. @YanchunHe , can you check and fix it?

@monsieuralok will need to republish this member after its fixed

YanchunHe commented 5 years ago

This is caused by conflict of json files when serval task runs for the same experiment (which has been fixed).

I will redo part of the cmorization for this realisation.

monsieuralok commented 5 years ago

Hi, Member 72 and 85 pdSST-futArcSIC and member 84 and 91 pdSST-pdSIC also seems to have problem. Please can you check it.

YanchunHe commented 5 years ago

Member 94,72,85 for pdSST-futArcSIC; member 84 and 91 for pdSST-pdSIC have been updated. sha256sum and QCreports have been updated.

Ready to publish to ESGF (refer Issue #37 )

monsieuralok commented 5 years ago

@YanchunHe Published

lisesg commented 5 years ago

Thanks a lot for CMOR-izing and publishing the first set of variables! I've just had a request for daily zg500 from someone working on a study on blocking, and it's not among the variables that have been CMOR-ized so far. Would it be possible to CMOR-ize zg500 or is it not supported yet?

YanchunHe commented 5 years ago

I see zg500 is in the AERday table, which is not supported yet. I am currently cmorization the experiment with current supported tables, and will work to add more fields soon later.

I foresee that I will include quite some more tables/fields in about two weeks. And then redo the cmorization for all finished experiments with these additionally supported fields with new data version. Will that be too late?

I don't want to add only a few variables and publish a new release. That will increase the workload for cmorization, publication a lot, considering so many experiments.

IngoBethke commented 5 years ago

Daily zg500 is a key variable for many atmospheric studies that focus on dynamics. If we wait a couple of weeks then we risk that NorESM2 will not contribute to community papers central to IPCC. Therefore I'd suggest to prioritise the implementation of AERday and subsequent cmor-ization of zg500 and not make this dependent on the cmor-ization of other tables and variables.

lisesg commented 5 years ago

It looks like a lot of the other MIPs and the DECK have requested the full 3-D daily zg, which includes the 500 hPa surface, and makes daily zg500 redundant. From the CMIP6 data request spreadsheet, it looks like PAMIP and PMIP are the only MIPs that explicitly ask for daily zg500 (unless I'm missing something), probably because daily zg is not a requested field for those MIPs.

Edit: some columns where for some reason not visible when I was searching the spreadsheet, looks like daily zg500 is also a priority 1 field for AerChemMIP and DCPP.

IngoBethke commented 5 years ago

I have added support for the tables Eday, CFday and AERday in the latest commit https://github.com/NorwegianClimateCentre/noresm2cmor/commit/66ed9776f2efbafbef8ec8f775a7007632b5b0a6

I have created a namelist specification for NorCPM1 to output selected daily variables https://github.com/NorwegianClimateCentre/noresm2cmor/blob/master/namelists/CMIP6_NorCPM1/dcppA-hindcast_i1_v20191005/var.nml (note that it uses the "old" column order)

YanchunHe commented 5 years ago

Great, thanks, I will later incorporate this to NorESM2.

IngoBethke commented 5 years ago

Note that I also fixed a bug (missing r

field in namelists) that would cause CMIP6 tables to skipped if e.g. rOday was specified.

I've just configured another daily variable TMQ -> prw to NorCPM1's daily output. NorCPM1 daily output variables I've so far configured that are not in table "day" are (old column order):

&table_AERday dAERday = .true. tAERday = 'CMIP6_AERday.json', rAERday = 1000000, vAERday = 'Z010 ','zg10 ',' ', 'Z500 ','zg500 ',' ', 'U010 ','ua10 ',' ', 'V010 ','va10 ',' ', /

&table_Eday dEday = .true. tEday = 'CMIP6_Eday.json', rEday = 1000000, vEday = 'TMQ ','prw ','kg m-2', 'PRECTMX ','prhmax ','kg m-2 s-1', 'T500 ','ta500 ',' ', 'T850 ','ta850 ',' ', /

&table_CFday dCFday = .true. tCFday = 'CMIP6_CFday.json', rCFday = 1000000, vCFday = 'OMEGA500 ','wap500 ',' ', /

&table_SIday dSIday = .true. tSIday = 'CMIP6_SIday.json', rSIday = 1000000, vSIday = 'aice_d ','siconc ',' ', 'hi_d ','sithick ',' ', 'fs_d ','sisnconc ',' ', 'uvel_d ','siu ','zero2missing', 'vvel_d ','siv ','zero2missing', /

&table_Oday dOday = .true. tOday = 'CMIP6_Oday.json', rOday = 1000000, vOday = 'sst ','tos ',' ', 'maxmld ','omldamax ',' ', /

YanchunHe commented 5 years ago

A bug in the namelist file (refer to #78), which cause identical cmorized output among different members/realisations for *day tables., has been fixed.

*day fields (including only zg500_AERday and ta850_Eday) have been recmorized, and tagged as v20191018

cmorized files are now ready to publish to ESGF, in order to replace the wrong v20191009 version.

data path

  • /projects/NS9034K/CMIP6/PAMIP/NCC/NorESM2-LM/pdSST-pdSIC
  • /projects/NS9034K/CMIP6/PAMIP/NCC/NorESM2-LM/pdSST-futArcSIC

version

  • v20191018 (all the 100 members)

other information

  • sha256sum updated
  • PrePARE QC passed

May need to retract the v20191009 data at some point.

I think this should be put with high-priority.

monsieuralok commented 5 years ago

@YanchunHe published

YanchunHe commented 5 years ago

cmorized with additional fields

data path

  • /projects/NS9034K/CMIP6/PAMIP/NCC/NorESM2-LM/pdSST-pdSIC
  • /projects/NS9034K/CMIP6/PAMIP/NCC/NorESM2-LM/pdSST-futArcSIC

version

  • v20191108 (all the 100 members)

sha256sum

  • .ri1p1f1.sha256sum_v20191108b ( include 1-100)
monsieuralok commented 4 years ago

@YanchunHe @IngoBethke @Kirkevag published

YanchunHe commented 3 months ago

Hi @monsieuralok

Could you please publish the v20191018 version datasets (they are by any means failed to be published, as discussed in the issue #78 )?

For corrected AERday datasets

data path

  • /projects/NS9034K/CMIP6/PAMIP/NCC/NorESM2-LM/pdSST-pdSIC
  • /projects/NS9034K/CMIP6/PAMIP/NCC/NorESM2-LM/pdSST-futArcSIC

version

  • v20191008 (all the 100 members)

sha256sum

  • .ri1p1f1.sha256sum_v20191008 ( include 1-100)

correction:, the dataset should be v20191018 (v20191008 is a typo).

monsieuralok commented 3 months ago

@YanchunHe is it version number v20191018 or v20191008? As, I have not found v20191008

YanchunHe commented 3 months ago

@YanchunHe is it version number v20191018 or v20191008? As, I have not found v20191008

You are right, it is v20191018.

monsieuralok commented 2 months ago

@YanchunHe published

lisesg commented 2 months ago

Is there an errata for the the issue described in #78?

YanchunHe commented 2 months ago

Is there an errata for the the issue described in #78?

If they were just not published, there was noting wrong appears in the ESGF node, so there should be no errata, right?

@monsieuralok How do you think?

lisesg commented 2 months ago

For the first 100 members of pdSST-pdSIC and pdSST-futArcSIC, the incorrect version of AERday zg500 was published in which multiple members had identical data as described in #78.

This does however not apply to members 101-200, as they were run and cmorized at a later time they were not affected by the same error.

YanchunHe commented 2 months ago

For the first 100 members of pdSST-pdSIC and pdSST-futArcSIC, the incorrect version of AERday zg500 was published in which multiple members had identical data as described in #78.

This does however not apply to members 101-200, as they were run and cmorized at a later time they were not affected by the same error.

OK, this has been a bit long, and I really don't quick figure out what the problem was 😅

Is it OK if you are able directly give the information to Alok what need to be reported, for which datasets (exp. realisation, variable names, etc)?

lisesg commented 2 months ago

OK, this has been a bit long, and I really don't quick figure out what the problem was 😅

Is it OK if you are able directly give the information to Alok what need to be reported, for which datasets (exp. realisation, variable names, etc)?

No problem, I will contact Alok. :)

monsieuralok commented 2 months ago

Opened Errata issue:- https://errata.ipsl.fr/static/view.html?uid=c9dc7541-a0e1-e06a-859b-ce4633bb2f8a

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.