search

NorESMhub / noresm2cmor

A command line tool for cmorizing NorESM output
http://noresmhub.github.io/noresm2cmor/
5 stars 16 forks source link

[CMIP6 CMOR-ization & ESGF-publication] NorESM2-MM - piControl #140

Closed matsbn closed 1 year ago

matsbn commented 4 years ago

Mandatory information:

Full path to the case(s) of the experiment on NIRD /projects/projects/NS9560K/noresm/cases /projects/projects/NS9560K/FRAM/noresm/cases

experiment_id piControl

model_id NorESM2-MM

CASENAME(s) and years to be CMORized N1850frc2_f09_tn14_20191001, 1200-1299 N1850frc2_f09_tn14_20191012, 1300-1449 N1850frc2_f09_tn14_20191113, 1450-1699

Optional information

parent_experiment_id piControl-spinup

parent_experiment_rip r1i1p1f1

parent_time_units 'days since 0001-01-01'

branch_method 'Hybrid-restart from year 1200-01-01 of piControl-spinup'

other information

jgriesfeller commented 4 years ago

I decided to put the 9.5Tb of N1850frc2_f09_tn14_20191113 NIRD at /tos-project3/NS9560K/noresm/cases omitting the rest folder. I will also remove data from /tos-project1/NS9034K/noresm/cases that has been cmorised and published. I think leaving that on NIRD is not an option since we need the space for cmorisation.

current quota status :

(base) [jang@login-nird-2 cases]$ dusage -p ns9034k
==========================================================================================
Project      Account    Resource        Type    Usage      Soft Limit   Hard Limit  
NS9034K      $PROJECT    nird           Disk    245.529TB    270TB        270TB                     
NS9034K      $PROJECT    nird           Files   3584082      5242880      5242880              
------------------------------------------------------------------------------------------
(base) [jang@login-nird-2 cases]$ dusage -p ns9560K
==========================================================================================
Project      Account    Resource        Type    Usage      Soft Limit   Hard Limit  
NS9560K      $PROJECT    nird           Disk    199.148TB    230TB        230TB                     
NS9560K      $PROJECT    nird           Files   4320007      6029312      6291456              
------------------------------------------------------------------------------------------
jgriesfeller commented 4 years ago

In order to use the computing power we have, I also started the 2nd faulty period (1410 to 1449). Host is login2, script name is cmor_tmp_p2.sh. I am aware that I have to run the quality control etc by hand at the end.

jgriesfeller commented 4 years ago

current status:

(base) [jang@login-nird-0 piControl]$ ./checkcmorout.sh 
real:   r1i1p1f1
Ofx, fx, etc    8
yyyy1   yyyy2   nf
1200    1209    517
1210    1219    517
1220    1229    517
1230    1239    517
1240    1249    517
1250    1259    517
1260    1269    517
1270    1279    517
1280    1289    517
1290    1299    517
1300    1309    581
1310    1319    581
1320    1329    581
1330    1339    581
1340    1349    581
1350    1359    209
1360    1369    581
1370    1379    193
1380    1389    581
1390    1399    581
1400    1409    581
1410    1419    544
1420    1429    145
1430    1439    113
1440    1449    115
1450    1450    581
Total:      12307

Just one complete period more than yesterday (1360 1369 581). The job for the period 1350 to 1370 died again.

jgriesfeller commented 4 years ago

N1850frc2_f09_tn14_20191113 has been successfully transferred to /projects/NS9560K/noresm/cases omitting the rest directory.

jgriesfeller commented 4 years ago

Current status (all jobs have died again):

(base) [jang@login-nird-0 piControl]$ ./checkcmorout.sh 
real:   r1i1p1f1
Ofx, fx, etc    8
yyyy1   yyyy2   nf
1200    1209    517
1210    1219    517
1220    1229    517
1230    1239    517
1240    1249    517
1250    1259    517
1260    1269    517
1270    1279    517
1280    1289    517
1290    1299    517
1300    1309    581
1310    1319    581
1320    1329    581
1330    1339    581
1340    1349    581
1350    1359    209
1360    1369    581
1370    1379    193
1380    1389    581
1390    1399    581
1400    1409    581
1410    1419    544
1420    1429    581
1430    1439    113
1440    1449    115
1450    1450    581
Total:      12743
Unfinished files:

So one additional period within one day with several periods failing after the same number of files. I am not sure if we can get this done with the current system if we cant just run the remaining variables.

YanchunHe commented 4 years ago

restarted

YanchunHe commented 4 years ago

status update:

piControl has been cmorized through years 1200-1560 except two periods 1410 and 1430. I will try to write a general script to pick out and only cmorize those unfinished fields. Jobs for years 1560 and onwards are running.

$ ./checkcmorout.sh real: r1i1p1f1 Ofx, fx, etc 8 yyyy1 yyyy2 nf 1200 1209 517 1210 1219 517 1220 1229 517 1230 1239 517 1240 1249 517 1250 1259 517 1260 1269 517 1270 1279 517 1280 1289 517 1290 1299 517 1300 1309 581 1310 1319 581 1320 1329 581 1330 1339 581 1340 1349 581 1350 1359 581 1360 1369 581 1370 1379 581 1380 1389 581 1390 1399 581 1400 1409 581 1410 1419 544 1420 1429 581 1430 1439 544 1440 1449 581 1450 1450 581 1450 1459 517 1460 1469 517 1470 1479 517 1480 1489 517 1490 1499 517 1500 1509 517 1510 1519 517 1520 1529 517 1530 1539 517 1540 1549 517 1550 1559 517 1560 1569 492 1570 1579 0 1580 1589 0 1590 1599 0 1600 1609 0 1610 1619 0 1620 1629 0 1630 1639 0 1640 1649 0 1650 1659 0 1660 1669 0 1670 1679 0 1680 1689 0 1690 1699 0 Total: 20579

YanchunHe commented 4 years ago

cmorized and ready to publish (4th of Jan).

data path

version

sha256sum

status: $ ./checkcmorout.sh real: r1i1p1f1 Ofx, fx, etc 8 yyyy1 yyyy2 nf 1200 1209 517 1210 1219 517 1220 1229 517 1230 1239 517 1240 1249 517 1250 1259 517 1260 1269 517 1270 1279 517 1280 1289 517 1290 1299 517 1300 1309 581 1310 1319 581 1320 1329 581 1330 1339 581 1340 1349 581 1350 1359 581 1360 1369 581 1370 1379 581 1380 1389 581 1390 1399 581 1400 1409 581 1410 1419 581 1420 1429 581 1430 1439 581 1440 1449 581 1450 1450 581 1450 1459 517 1460 1469 517 1470 1479 517 1480 1489 517 1490 1499 517 1500 1509 517 1510 1519 517 1520 1529 517 1530 1539 517 1540 1549 517 1550 1559 517 1560 1569 517 1570 1579 517 1580 1589 517 1590 1599 517 1600 1609 517 1610 1619 517 1620 1629 517 1630 1639 517 1640 1649 517 1650 1659 517 1660 1669 517 1670 1679 517 1680 1689 517 1690 1699 517 Total: 27399

monsieuralok commented 4 years ago

@YanchunHe published but, presently let this issue open

YanchunHe commented 4 years ago

(refer to M. Stolpe). year 1450 seems to appear twice in the NorESM2-MM piControl simulation, in these two files:

tas_Amon_NorESM2-MM_piControl_r1i1p1f1_gn_145001-145012.nc tas_Amon_NorESM2-MM_piControl_r1i1p1f1_gn_145001-145912.nc (http://noresg.nird.sigma2.no/thredds/catalog/esgcet/779/CMIP6.CMIP.NCC.NorESM2-MM.piControl.r1i1p1f1.Amon.tas.gn.v20191108.html#CMIP6.CMIP.NCC.NorESM2-MM.piControl.r1i1p1f1.Amon.tas.gn.v20191108)

This issue is not limited to the variable 'tas', but also for other variables (but I haven't checked all). The smaller, one year file appears to be older than the decadal file.

YanchunHe commented 4 years ago

(refer to M.Bentsen)

I agree this is confusing. The case N1850frc2_f09_tn14_20191012 that ends after 1450 is with high frequency output, while N1850frc2_f09_tn14_20191113 that starts at the beginning of 1450 is with normal output. Year 1450 of piControl corresponds to year 2100 of the scenario integrations, all with high frequency output, and we should keep the single year files of piControl for variables only available with high frequency. Single year files that are duplicate I think should be retracted. Would this be possible?

YanchunHe commented 4 years ago

Hi Alok,

I have identified the files of normal frequency that appears in both single 1450 year and in 1450-1459. You can find the full list of files at: /tos-project1/NS9034K/CMIP6/.cmorout/NorESM2-MM/piControl/1450_duplicated.txt

Could you manage to retract these files from ESGF? Thanks!

@monsieuralok

monsieuralok commented 4 years ago

@YanchunHe It is not possible to takeout a single file from dataset; I have to retract whole dataset. But, I will again check it. @IngoBethke Could you explain if is it possible to retract a single file from dataset?

YanchunHe commented 4 years ago

Alok has retracted the whole dataset. I have updated the linked files following the ESGF folder structure, and update the hash files.

Please republish the whole experiment with information from the updated hash file: /projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-MM/piControl/.r1i1p1f1.sha256sum_v20191108

@monsieuralok

monsieuralok commented 4 years ago

@matsbn @YanchunHe published

YanchunHe commented 3 years ago

Cmorized with additional iLAMB variables (#262), AERday zg500 (#263) and corrected fNup (#251).

They are ready to be published to ESGF.

data path

version

sha256sum /projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-MM/piControl

monsieuralok commented 3 years ago

@YanchunHe published

YanchunHe commented 3 years ago

New dataset version to fix issues #269, #270, #271, #272, #273 is ready to be published:

data path

version

sha256sum /projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-MM/piControl

Note, 6 hourly data are only available for years 1300-1450.

monsieuralok commented 3 years ago

@YanchunHe published

monsieuralok commented 1 year ago

@YanchunHe retracted

YanchunHe commented 1 year ago

Seems like there are no 3 hourly precipitation data available in the model output, so no cmorization will done for MM piControl as discussed in issue #41

YanchunHe commented 1 year ago

No, some years has 3 hourly precipitation in MM piControl.

YanchunHe commented 1 year ago

CMORized additional 3-hourly precipitation dataset for NorESM2-MM piControl, only for years 1300-1309.

data path

version

sha256sum /projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-MM/piControl

monsieuralok commented 1 year ago

@YanchunHe published