search

payu-org / payu

A workflow management tool for numerical models on the NCI computing systems
Apache License 2.0
18 stars 25 forks source link

ACCESS-OM2: OverflowError: date value out of range #303

Closed hakaseh closed 3 years ago

hakaseh commented 3 years ago

Here I am reposting my old issue (previously I posted on access-om2): https://github.com/COSIMA/access-om2/issues/228

My run is at Year 2501, which does not start due to the following error:

vi /scratch/v45/hh0162/access-om2/control/1deg_jra55_iaf_omip2r1i2p2f1.e14870883

Traceback (most recent call last): File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-20.07/bin/payu-run", line 10, in sys.exit(runscript()) File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-20.07/lib/python3.7/site-packages/payu/subcommands/run_cmd.py", line 131, in runscript expt.setup() File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-20.07/lib/python3.7/site-packages/payu/experiment.py", line 422, in setup model.setup() File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-20.07/lib/python3.7/site-packages/payu/models/cice5.py", line 56, in setup super(Cice5, self).setup() File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-20.07/lib/python3.7/site-packages/payu/models/cice.py", line 217, in setup caltype) File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-20.07/lib/python3.7/site-packages/payu/calendar.py", line 51, in date_plus_seconds end_date += get_leapdays(init_date, end_date) OverflowError: date value out of range

Any idea why this happens? My latest successful output and restart files are labeled 2499 (output2499 and restart2499).

aidanheerdegen commented 3 years ago

That path doesn't exist:

$ ls -l /scratch/v45/hh0162/access-om2/control/1deg_jra55_iaf_omip2r1i2p2f1.e14870883
ls: cannot access '/scratch/v45/hh0162/access-om2/control/1deg_jra55_iaf_omip2r1i2p2f1.e14870883': No such file or directory

Can you check that and get back to me with the path to the control directory and error file.

aidanheerdegen commented 3 years ago

@hakaseh I can't look at this unless you provide the information I've asked for above.

hakaseh commented 3 years ago

@aidanheerdegen sorry for the late reply. I just re-ran the model and reproduced the issue:

/scratch/v45/hh0162/access-om2/control/1deg_jra55_iaf_omip2/r1i2p2f1.e23212195

thanks for looking into this. i'd love to continue the run beyond 2500 years, so it would be great if this gets fixed.

hakaseh commented 3 years ago

thanks @aidanheerdegen for https://github.com/payu-org/payu/pull/304#issuecomment-852661754 how can i update payu? i don't know where it is located.

aidanheerdegen commented 3 years ago

There is an issue with updating the conda environment. I'll let you know when that is fixed.

aidanheerdegen commented 3 years ago

payu/1.0.21 is now installed in conda/analysis3-unstable (conda/analysis3-21.04)

You can check you've got the correct conda environment loaded with the latest version with the conda list command:

$ conda list payu
# packages in environment at /g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.04:
#
# Name                    Version                   Build  Channel
payu                      1.0.21                     py_0    coecms

I have tested with your configuration, but let me know if that doesn't work for you @hakaseh

hakaseh commented 3 years ago

@aidanheerdegen i am happy to confirm that my test run was successful with the payu update. thank you!