search

esm-tools / esm_tools

Simple Infrastructure for Earth System Simulations
https://esm-tools.github.io/
GNU General Public License v2.0
25 stars 12 forks source link

work folder in run_* is not properly cleaned up after a run #55

Closed christian-stepanek closed 4 years ago

christian-stepanek commented 4 years ago

Expected behavior: After completing a run (here: one model year) the model output and restart files should be moved from run????0101-????1231/work/ to the respective "permanent" locations in experiment subfolders {outdata,restart}/{echam,fesom,hdmodel,jsbach,oasis3mct} in order to leave a clean folder structure. In run* only those files should remain that cannot be sensibly associated with any of the "permanent" locations (if there are any such files).

Observed behavior: The expected behavior is only partially completed by the scripts (see screen listing at the end of this issue report). As a result the tools leave behind bloated up run_ folders while the simulation output is only partially available at the expected location (outdata/). Interestingly, the behavior is different for the first year than for the following years:

stepanek@ollie0:/work/ollie/stepanek/esm-experiments_v4/myinitialtest_yearly$ du -sh *
512 analysis
52M bin
393K    config
512 couple
1,2G    forcing
126M    input
710M    log
512 mon
155G    outdata
3,1G    restart
161G    run_20000101-20001231
11G run_20010101-20011231
12G run_20020101-20021231
1,6M    scripts
0   unknown
512 viz
0   work

Large file size of outdata stems from the oasis3mct subfolder - is the respective data intended to be there?

stepanek@ollie0:/work/ollie/stepanek/esm-experiments_v4/myinitialtest_yearly$ du -sh outdata/*
1,1G    outdata/echam
2,2G    outdata/fesom
0   outdata/hdmodel
6,5M    outdata/jsbach
153G    outdata/oasis3mct
stepanek@ollie0:/work/ollie/stepanek/esm-experiments_v4/myinitialtest_yearly$ du -sh outdata/oasis3mct/*
1,3G    outdata/oasis3mct/evap_atm_echam6_06.nc
8,4G    outdata/oasis3mct/evap_oce_fesom_06.nc
1,3G    outdata/oasis3mct/heat_atm_echam6_08.nc
1,3G    outdata/oasis3mct/heat_ica_echam6_09.nc
8,4G    outdata/oasis3mct/heat_ico_fesom_09.nc
8,4G    outdata/oasis3mct/heat_oce_fesom_08.nc
1,3G    outdata/oasis3mct/heat_swa_echam6_10.nc
8,4G    outdata/oasis3mct/heat_swo_fesom_10.nc
1,3G    outdata/oasis3mct/hydr_atm_echam6_11.nc
8,4G    outdata/oasis3mct/hydr_oce_fesom_11.nc
1,3G    outdata/oasis3mct/prec_atm_echam6_04.nc
8,4G    outdata/oasis3mct/prec_oce_fesom_04.nc
1,3G    outdata/oasis3mct/sie_atmo_echam6_01.nc
8,4G    outdata/oasis3mct/sie_feom_fesom_01.nc
1,3G    outdata/oasis3mct/sit_atmo_echam6_01.nc
8,4G    outdata/oasis3mct/sit_feom_fesom_01.nc
1,3G    outdata/oasis3mct/snow_atm_echam6_05.nc
8,4G    outdata/oasis3mct/snow_oce_fesom_05.nc
1,3G    outdata/oasis3mct/snt_atmo_echam6_02.nc
8,4G    outdata/oasis3mct/snt_feom_fesom_02.nc
1,3G    outdata/oasis3mct/sst_atmo_echam6_01.nc
8,4G    outdata/oasis3mct/sst_feom_fesom_01.nc
1,3G    outdata/oasis3mct/subl_atm_echam6_07.nc
8,4G    outdata/oasis3mct/subl_oce_fesom_07.nc
1,3G    outdata/oasis3mct/taux_atm_echam6_03.nc
1,3G    outdata/oasis3mct/taux_ica_echam6_03.nc
8,4G    outdata/oasis3mct/taux_ico_fesom_03.nc
8,4G    outdata/oasis3mct/taux_oce_fesom_03.nc
1,3G    outdata/oasis3mct/tauy_atm_echam6_03.nc
1,3G    outdata/oasis3mct/tauy_ica_echam6_03.nc
8,4G    outdata/oasis3mct/tauy_ico_fesom_03.nc
8,4G    outdata/oasis3mct/tauy_oce_fesom_03.nc

In summary: It is not clear why some of the files remain in the run_* folders rather than being moved to outdata, and even more unclear is why the behavior is different in the first year. Furthermore, the meaning of the relatively large files in outdata/oasis3mct is not clear to me at the moment, and it must be checked whether these files should be there / need to be kept.

employed run script: /home/ollie/stepanek/esm-tools_v4/esm_tools/myrunscripts/myinitialtest_yearly.yaml

employed esm_tools versions:

esm_archiving : unknown version! esm_autotests : unknown version! esm_calendar : 4.0.1 esm_database : 4.0.0 esm_environment : 4.0.1 esm_master : 4.0.2 esm_parser : 4.0.2 esm_profile : 4.0.0 esm_rcfile : 4.0.0 esm_runscripts : 4.0.3 esm_tools : 4.0.9 esm_plugin_manager : 4.0.1 esm_version_checker : 4.0.2

diff between fesom output in the workfolder within run_... and the respective outdata folder:

stepanek@ollie0:/work/ollie/stepanek/esm-experiments_v4/myinitialtest_yearly$ diff <(cd outdata/fesom/; ls -1 *fesom.2000.*) <(cd run_20000101-20001231/work/; ls -1 *fesom*)
6a7
> Av.fesom.2000.nc
8a10,12
> bolus_u.fesom.2000.nc
> bolus_v.fesom.2000.nc
> bolus_w.fesom.2000.nc
9a14
> evap_oce_fesom_06.nc
11a17,21
> fesom
> fesom.2000.ice.restart.nc
> fesom.2000.oce.restart.nc
> fesom.clock
> fesom.mesh.diag.nc
14a25,28
> heat_ico_fesom_09.nc
> heat_oce_fesom_08.nc
> heat_swo_fesom_10.nc
> hydr_oce_fesom_11.nc
16a31
> Kv.fesom.2000.nc
20a36
> N2.fesom.2000.nc
21a38
> prec_oce_fesom_04.nc
23a41,50
> runoff.fesom.2000.nc
> salt.fesom.2000.nc
> sie_feom_fesom_01.nc
> sit_feom_fesom_01.nc
> slope_x.fesom.2000.nc
> slope_y.fesom.2000.nc
> slope_z.fesom.2000.nc
> snow.fesom.2000.nc
> snow_oce_fesom_05.nc
> snt_feom_fesom_02.nc
25a53
> sst_feom_fesom_01.nc
26a55,61
> subli.fesom.2000.nc
> subl_oce_fesom_07.nc
> taux_ico_fesom_03.nc
> taux_oce_fesom_03.nc
> tauy_ico_fesom_03.nc
> tauy_oce_fesom_03.nc
> temp.fesom.2000.nc
28a64
> u.fesom.2000.nc
29a66
> v.fesom.2000.nc
31a69
> w.fesom.2000.nc
stepanek@ollie0:/work/ollie/stepanek/esm-experiments_v4/myinitialtest_yearly$ diff <(cd outdata/fesom/; ls -1 *fesom.2001.*) <(cd run_20010101-20011231/work/; ls -1 *fesom*)
6a7
> Av.fesom.2001.nc
8a10,12
> bolus_u.fesom.2001.nc
> bolus_v.fesom.2001.nc
> bolus_w.fesom.2001.nc
11a16,21
> fesom
> fesom.2000.ice.restart.nc
> fesom.2000.oce.restart.nc
> fesom.2001.ice.restart.nc
> fesom.2001.oce.restart.nc
> fesom.clock
16a27
> Kv.fesom.2001.nc
20a32
> N2.fesom.2001.nc
23a36,41
> runoff.fesom.2001.nc
> salt.fesom.2001.nc
> slope_x.fesom.2001.nc
> slope_y.fesom.2001.nc
> slope_z.fesom.2001.nc
> snow.fesom.2001.nc
26a45,46
> subli.fesom.2001.nc
> temp.fesom.2001.nc
28a49
> u.fesom.2001.nc
29a51
> v.fesom.2001.nc
31a54
> w.fesom.2001.nc

stepanek@ollie0:/work/ollie/stepanek/esm-experiments_v4/myinitialtest_yearly$ ls -l outdata/fesom/*fesom*.2000.* | wc -l
31
stepanek@ollie0:/work/ollie/stepanek/esm-experiments_v4/myinitialtest_yearly$ ls -l run_20000101-20001231/work/*fesom*.2000.* | wc -l
50
stepanek@ollie0:/work/ollie/stepanek/esm-experiments_v4/myinitialtest_yearly$ ls -l outdata/fesom/*fesom*.2001.* | wc -l
31
stepanek@ollie0:/work/ollie/stepanek/esm-experiments_v4/myinitialtest_yearly$ ls -l run_20010101-20011231/work/*fesom*.2001.* | wc -l
50
christian-stepanek commented 4 years ago

Here is a list of files that are left in work after a run finishes, together with a tentative suggestion where these should be moved.

If somebody of the core developers pointed me to the esm-function, where the list of files to be moved from work to restart and outdata can be adjusted, I can look into a solution myself.

ocean output -> move from run_????0101-????1231/work to outdata/fesom
    a_ice.fesom.2179.nc
    alpha.fesom.2179.nc
    atmice_x.fesom.2179.nc
    atmice_y.fesom.2179.nc
    atmoce_x.fesom.2179.nc
    atmoce_y.fesom.2179.nc
    Av.fesom.2179.nc
    beta.fesom.2179.nc
    Bo.fesom.2179.nc
    bolus_u.fesom.2179.nc
    bolus_v.fesom.2179.nc
    bolus_w.fesom.2179.nc
    evap.fesom.2179.nc
    fer_C.fesom.2179.nc
    fer_K.fesom.2179.nc
    fesom.2178.ice.restart.nc
    fesom.2178.oce.restart.nc
    fesom.2179.ice.restart.nc
    fesom.2179.oce.restart.nc
    fh.fesom.2179.nc
    fw.fesom.2179.nc
        hbl.fesom.2179.nc
    iceoce_x.fesom.2179.nc
    iceoce_y.fesom.2179.nc
    Kv.fesom.2179.nc
    m_ice.fesom.2179.nc
    MLD1.fesom.2179.nc
    MLD2.fesom.2179.nc
    m_snow.fesom.2179.nc
    N2.fesom.2179.nc
    prec.fesom.2179.nc
    Redi_K.fesom.2179.nc
    reso.fesom.2179.nc
    runoff.fesom.2179.nc
    salt.fesom.2179.nc
    slope_x.fesom.2179.nc
    slope_y.fesom.2179.nc
    slope_z.fesom.2179.nc
    snow.fesom.2179.nc
    ssh.fesom.2179.nc
    sss.fesom.2179.nc
    sst.fesom.2179.nc
    subli.fesom.2179.nc
    temp.fesom.2179.nc
    tx_sur.fesom.2179.nc
    ty_sur.fesom.2179.nc
    u.fesom.2179.nc
    uice.fesom.2179.nc
    v.fesom.2179.nc
    vice.fesom.2179.nc
    vve.fesom.2179.nc
    w.fesom.2179.nc

atmosphere/land output -> move from run_????0101-????1231/work to outdata/{echam,jsbach}
    PI_ctrl_AWI-ESM-2-1_217901.01_accw
    PI_ctrl_AWI-ESM-2-1_217901.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217901.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217901.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217901.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217901.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217901.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_glday
    PI_ctrl_AWI-ESM-2-1_217901.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_glim
    PI_ctrl_AWI-ESM-2-1_217901.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217901.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217901.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217901.01_spim
    PI_ctrl_AWI-ESM-2-1_217901.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_accw
    PI_ctrl_AWI-ESM-2-1_217902.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217902.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217902.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217902.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217902.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217902.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_glday
    PI_ctrl_AWI-ESM-2-1_217902.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_glim
    PI_ctrl_AWI-ESM-2-1_217902.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217902.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217902.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217902.01_spim
    PI_ctrl_AWI-ESM-2-1_217902.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_accw
    PI_ctrl_AWI-ESM-2-1_217903.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217903.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217903.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217903.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217903.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217903.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_glday
    PI_ctrl_AWI-ESM-2-1_217903.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_glim
    PI_ctrl_AWI-ESM-2-1_217903.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217903.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217903.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217903.01_spim
    PI_ctrl_AWI-ESM-2-1_217903.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_accw
    PI_ctrl_AWI-ESM-2-1_217904.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217904.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217904.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217904.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217904.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217904.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_glday
    PI_ctrl_AWI-ESM-2-1_217904.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_glim
    PI_ctrl_AWI-ESM-2-1_217904.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217904.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217904.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217904.01_spim
    PI_ctrl_AWI-ESM-2-1_217904.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_accw
    PI_ctrl_AWI-ESM-2-1_217905.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217905.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217905.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217905.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217905.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217905.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_glday
    PI_ctrl_AWI-ESM-2-1_217905.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_glim
    PI_ctrl_AWI-ESM-2-1_217905.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217905.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217905.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217905.01_spim
    PI_ctrl_AWI-ESM-2-1_217905.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_accw
    PI_ctrl_AWI-ESM-2-1_217906.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217906.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217906.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217906.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217906.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217906.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_glday
    PI_ctrl_AWI-ESM-2-1_217906.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_glim
    PI_ctrl_AWI-ESM-2-1_217906.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217906.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217906.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217906.01_spim
    PI_ctrl_AWI-ESM-2-1_217906.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_accw
    PI_ctrl_AWI-ESM-2-1_217907.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217907.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217907.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217907.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217907.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217907.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_glday
    PI_ctrl_AWI-ESM-2-1_217907.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_glim
    PI_ctrl_AWI-ESM-2-1_217907.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217907.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217907.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217907.01_spim
    PI_ctrl_AWI-ESM-2-1_217907.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_accw
    PI_ctrl_AWI-ESM-2-1_217908.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217908.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217908.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217908.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217908.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217908.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_glday
    PI_ctrl_AWI-ESM-2-1_217908.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_glim
    PI_ctrl_AWI-ESM-2-1_217908.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217908.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217908.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217908.01_spim
    PI_ctrl_AWI-ESM-2-1_217908.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_accw
    PI_ctrl_AWI-ESM-2-1_217909.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217909.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217909.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217909.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217909.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217909.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_glday
    PI_ctrl_AWI-ESM-2-1_217909.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_glim
    PI_ctrl_AWI-ESM-2-1_217909.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217909.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217909.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217909.01_spim
    PI_ctrl_AWI-ESM-2-1_217909.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_accw
    PI_ctrl_AWI-ESM-2-1_217910.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217910.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217910.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217910.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217910.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217910.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_glday
    PI_ctrl_AWI-ESM-2-1_217910.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_glim
    PI_ctrl_AWI-ESM-2-1_217910.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217910.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217910.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217910.01_spim
    PI_ctrl_AWI-ESM-2-1_217910.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_accw
    PI_ctrl_AWI-ESM-2-1_217911.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217911.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217911.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217911.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217911.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217911.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_glday
    PI_ctrl_AWI-ESM-2-1_217911.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_glim
    PI_ctrl_AWI-ESM-2-1_217911.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217911.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217911.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217911.01_spim
    PI_ctrl_AWI-ESM-2-1_217911.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_accw
    PI_ctrl_AWI-ESM-2-1_217912.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_aclcim
    PI_ctrl_AWI-ESM-2-1_217912.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_217912.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_g3bday
    PI_ctrl_AWI-ESM-2-1_217912.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_g3bid
    PI_ctrl_AWI-ESM-2-1_217912.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_g3bim
    PI_ctrl_AWI-ESM-2-1_217912.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_glday
    PI_ctrl_AWI-ESM-2-1_217912.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_glim
    PI_ctrl_AWI-ESM-2-1_217912.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_jsbid
    PI_ctrl_AWI-ESM-2-1_217912.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_sp6h
    PI_ctrl_AWI-ESM-2-1_217912.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_217912.01_spim
    PI_ctrl_AWI-ESM-2-1_217912.01_spim.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_accw
    PI_ctrl_AWI-ESM-2-1_218001.01_accw.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_aclcim
    PI_ctrl_AWI-ESM-2-1_218001.01_aclcim.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_g3b1hi
    PI_ctrl_AWI-ESM-2-1_218001.01_g3b1hi.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_g3bday
    PI_ctrl_AWI-ESM-2-1_218001.01_g3bday.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_g3bid
    PI_ctrl_AWI-ESM-2-1_218001.01_g3bid.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_g3bim
    PI_ctrl_AWI-ESM-2-1_218001.01_g3bim.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_glday
    PI_ctrl_AWI-ESM-2-1_218001.01_glday.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_glim
    PI_ctrl_AWI-ESM-2-1_218001.01_glim.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_jsbid
    PI_ctrl_AWI-ESM-2-1_218001.01_jsbid.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_sp6h
    PI_ctrl_AWI-ESM-2-1_218001.01_sp6h.codes
    PI_ctrl_AWI-ESM-2-1_218001.01_spim
    PI_ctrl_AWI-ESM-2-1_218001.01_spim.codes

diagnostic output including logs, executables, input, forcing, ...? -> delete? move to log? move to somewhere else?
    aero_coarse_2178.nc
    aero_coarse_2179.nc
    aero_coarse_2180.nc
    aero_farir_2178.nc
    aero_farir_2179.nc
    aero_farir_2180.nc
    aero_fine_2178.nc
    aero_fine_2179.nc
    aero_fine_2180.nc
    a_nd_file.nc
    areas.nc
    a2o_flux
    atmout
    cf_name_table.txt
    debug_notroot.01
    debug_notroot.02
    debug.root.01
    debug.root.02
    echam6
    ECHAM6_CldOptProps.nc
    echam.stderr
    echam.stdout
    fesom
    fesom.clock
    grids.nc
    hdpara.nc
    hdrestart.nc ??? is this the old or the new restart?
    hdstart.nc
    hostfile_srun
    jsbach.nc
    landuse_harvest.2178.nc
    landuse_harvest.2179.nc
    landuse_harvest.2180.nc
    landuseTransitions.2178.nc
    landuseTransitions.2179.nc
    landuseTransitions.2180.nc
    lctlib.def
    lightning.nc
    masks.nc
    namcouple
    namelist.config
    namelist.diag
    namelist.echam
    namelist.forcing
    namelist.ice
    namelist.io
    namelist.jsbach
    namelist.oce
    Ndepo.2178.nc
    Ndepo.2179.nc
    Ndepo.2180.nc
    nout.000000
    o2a_flux
    ozon2178
    ozon2179
    ozon2180
    rmp_atmo_to_feom_BICUBIC.nc
    rmp_atmo_to_feom_DISTWGT.nc
    rmp_feom_to_atmo_DISTWGT.nc
    rmp_hd.nc
    rrtmg_lw.nc
    rrtmg_sw.nc
    unit.23
    unit.24
    unit.90
    unit.91
    unit.92
    population_density.nc

restarts -> move in an orderly way to restart/{echam,fesom,hdmodel,jsbach,oasis3mct}
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_accw.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_aclcim.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_co2.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_echam.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_g3b1hi.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_g3bday.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_g3bid.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_g3bim.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_glday.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_glim.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_jsbach.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_jsbid.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_sp6h.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_spim.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_surf.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_veg.nc
    restart_PI_ctrl_AWI-ESM-2-1_21791231235230_yasso.nc
    restart_PI_ctrl_AWI-ESM-2-1_accw.nc
    restart_PI_ctrl_AWI-ESM-2-1_accw.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_aclcim.nc
    restart_PI_ctrl_AWI-ESM-2-1_aclcim.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_co2.nc
    restart_PI_ctrl_AWI-ESM-2-1_co2.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_echam.nc
    restart_PI_ctrl_AWI-ESM-2-1_echam.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_g3b1hi.nc
    restart_PI_ctrl_AWI-ESM-2-1_g3b1hi.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_g3bday.nc
    restart_PI_ctrl_AWI-ESM-2-1_g3bday.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_g3bid.nc
    restart_PI_ctrl_AWI-ESM-2-1_g3bid.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_g3bim.nc
    restart_PI_ctrl_AWI-ESM-2-1_g3bim.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_glday.nc
    restart_PI_ctrl_AWI-ESM-2-1_glday.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_glim.nc
    restart_PI_ctrl_AWI-ESM-2-1_glim.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_jsbach.nc
    restart_PI_ctrl_AWI-ESM-2-1_jsbach.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_jsbid.nc
    restart_PI_ctrl_AWI-ESM-2-1_jsbid.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_sp6h.nc
    restart_PI_ctrl_AWI-ESM-2-1_sp6h.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_spim.nc
    restart_PI_ctrl_AWI-ESM-2-1_spim.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_surf.nc
    restart_PI_ctrl_AWI-ESM-2-1_surf.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_veg.nc
    restart_PI_ctrl_AWI-ESM-2-1_veg.nc.bak
    restart_PI_ctrl_AWI-ESM-2-1_yasso.nc
    restart_PI_ctrl_AWI-ESM-2-1_yasso.nc.bak

just delete?
blabla:
christian-stepanek commented 4 years ago

Another big junk of data is left at the restart folder inside the run folder:

stepanek@ollie0:/work/ollie/abose/esm-experiments/PI_ctrl_AWI-ESM-2-1/run_21760101-21761231$ ls restart/*
restart/echam:

restart/fesom:
fesom.2175.ice.restart.nc  fesom.2175.oce.restart.nc

restart/hdmodel:
hdrestart.nc  work

restart/jsbach:

restart/oasis3mct:
a2o_flux  grids.nc  o2a_flux                     rmp_atmo_to_feom_DISTWGT.nc
areas.nc  masks.nc  rmp_atmo_to_feom_BICUBIC.nc  rmp_feom_to_atmo_DISTWGT.nc
stepanek@ollie0:/work/ollie/abose/esm-experiments/PI_ctrl_AWI-ESM-2-1/run_21760101-21761231

Files in restart/fesom could IMHO just get moved to EXPID/restart/fesom, am I right? The files are actually moved to restart, so maybe it is just forgotten to clean these up here? One has to think about what to do with output in restart/oasis3mct; if these are to be moved to EXPID/restart/oasis3, then it must be sure that they receive a proper time stamp in their name. Similar for files in restart/hdmodel.

pgierz commented 4 years ago

Closing in favour of #96