Error occurred while using the restart file generated by global simulation in nested simulation

[IreneAtmos]

Your name

Irene

Your affiliation

Southern University of Science and Technology

What happened? What did you expect to happen?

What happened?

• I ran a global 2x2.5 simulation for 2019 (2019global) and used the output restart files for September as the initial condition for a 2x2.5 simulation for September 2019 (09global) and 0.25x0.3125 nested simulation (09nested).
• This output restart file successfully drove the 2x2.5 simulation 09global .
• However, the 0.25x0.3125 nested simulation 09nested unexpectedly crashed at the start when using this output restart file. Only when I use the restart file from the GEOS-Chem benchmark directory, the simulation runs smoothly.

What did you expect to happen?

• I expected to be able to use the output restart file from 2019global to successfully start the 09nested simulation without any interruptions.
• Additionally, I hope to clarify the differences between the output restart file from 2019global and the benchmark restart file, and to understand why the output restart file from 2019global simulation drives the 09global simulation without interruption but causes the 09nested simulation to crash.
• Finally, I would appreciate suggestions on which restart file would be more appropriate for my 09nested simulation.

What are the steps to reproduce the bug?

1. Run the 2019 Global Simulation2019global:
   • Configure and execute the global simulation for the year 2019 at a 2x2.5 resolution, ensuring it completes successfully and generates the September restart file output.
2. Set Up the Nested Simulation09nested:
   • Configure the input files for the 0.25x0.3125 nested simulation, and copy the September restart file output in run directory.
3. Run the 2019Sep Global Simulation09global:
   • The normal output can be found in 09nested_log.0.

     
     HEMCO WARNING: Data is treated as unitless, but file attribute suggests it is not: cm3(H2O) cm-3(air). File: ./Restarts/GEOSChem.Restart.20190901_0000z.nc4
     --> LOCATION: HCOIO_READ (HCOIO_READ_STD_MOD.F90)
     TIMEZONES (i.e. OFFSETS FROM UTC) WERE READ FROM A FILE
     - Found all CN     met fields for 2011/01/01 00:00
     - Found all A1     met fields for 2019/09/01 00:30
     - Found all A3cld  met fields for 2019/09/01 01:30
     - Found all A3dyn  met fields for 2019/09/01 01:30
     - Found all A3mstC met fields for 2019/09/01 01:30
     - Found all A3mstE met fields for 2019/09/01 01:30
     - Found all I3     met fields for 2019/09/01 00:00
     Initialize DELP_DRY from restart file
     - Found all I3     met fields for 2019/09/01 03:00
     ===============================================================================
     R E S T A R T   F I L E   I N P U T

Min and Max of each species in restart file [mol/mol]: Species 1, ACET: Min = 8.681771098E-26 Max = 9.508711152E-09 Sum = 2.981275902E-04 ......

4.  Run the Nested Simulation and Find it Stops: 
    - The final output when the simulation terminates can be found in `09nested_log.0`.

HEMCO WARNING: Data is treated as unitless, but file attribute suggests it is not: cm3(H2O) cm-3(air). File: ./Restarts/GEOSChem.Restart.20190901_0000z.nc4 --> LOCATION: HCOIO_READ (HCOIO_READ_STD_MOD.F90)

### Please attach any relevant configuration and log files.

[09nested_geoschem_config.yml.txt](https://github.com/user-attachments/files/17521487/09nested_geoschem_config.yml.txt)
[09nested_HEMCO_Config.rc.txt](https://github.com/user-attachments/files/17521488/09nested_HEMCO_Config.rc.txt)
[09nested_HISTORY.rc.txt](https://github.com/user-attachments/files/17521489/09nested_HISTORY.rc.txt)
[09nested_log.0.txt](https://github.com/user-attachments/files/17521490/09nested_log.0.txt)
[09nested_log.e.txt](https://github.com/user-attachments/files/17521491/09nested_log.e.txt)
[09nested_log.o.txt](https://github.com/user-attachments/files/17521492/09nested_log.o.txt)
[2019global_geoschem_config.yml.txt](https://github.com/user-attachments/files/17521493/2019global_geoschem_config.yml.txt)
[2019global_HEMCO_Config.rc.txt](https://github.com/user-attachments/files/17521495/2019global_HEMCO_Config.rc.txt)
[2019global_HISTORY.rc.txt](https://github.com/user-attachments/files/17521496/2019global_HISTORY.rc.txt)
[2019global_log.0.txt](https://github.com/user-attachments/files/17521497/2019global_log.0.txt)
[2019global_log.o.txt](https://github.com/user-attachments/files/17521500/2019global_log.o.txt)
[09global_geoschem_config.yml.txt](https://github.com/user-attachments/files/17521481/09global_geoschem_config.yml.txt)
[09global_HEMCO_Config.rc.txt](https://github.com/user-attachments/files/17521482/09global_HEMCO_Config.rc.txt)
[09global_HISTORY.rc.txt](https://github.com/user-attachments/files/17521483/09global_HISTORY.rc.txt)
[09global_log.0.txt](https://github.com/user-attachments/files/17521484/09global_log.0.txt)
[09global_log.o.txt](https://github.com/user-attachments/files/17521486/09global_log.o.txt)

### What GEOS-Chem version were you using?

14.0.1

### What environment were you running GEOS-Chem on?

Local cluster

### What compiler and version were you using?

gcc 4.8.5, ifort 18.0.5

### Will you be addressing this bug yourself?

Yes, but I will need some help

### In what configuration were you running GEOS-Chem?

GCClassic

### What simulation were you running?

Full chemistry

### As what resolution were you running GEOS-Chem?

0.25x0.3125 nested ( 10N~55N, 70E~140E)

### What meterology fields did you use?

GEOS-FP

### Additional information

If using full chemistry, specify extra options like benchmark, complexSOA, etc. in the additional information section below:
Simulation type: `Full chemistry`
Additional simulation option: `Standard`

[yantosca]

Thanks for writing @IreneAtmos. I think the reason the nested run crashed is that you may be using a restart file without all o the species in it. Try changing this line in your nested-grid HEMCO_Config.rc from:

* SPC_           ./Restarts/GEOSChem.Restart.$YYYY$MM$DD_$HH$MNz.nc4 SpeciesRst_?ALL?    $YYYY/$MM/$DD/$HH EFYO xyz 1 * - 1 1

to

* SPC_           ./Restarts/GEOSChem.Restart.$YYYY$MM$DD_$HH$MNz.nc4 SpeciesRst_?ALL?    $YYYY/$MM/$DD/$HH EY xyz 1 * - 1 1

and see if that gets past the problem.

Long story short, the EFYO option will halt if there are species in your simulation that are not found in the restart file. EY will cause the simulation to proceed (and will assign a default background concentration to the missing species). You can also try CYS if that doesn't work.

For more information, please see:

• https://geos-chem.readthedocs.io/en/stable/gcclassic-user-guide/restart-files-gc.html

[IreneAtmos]

@yantosca Thank you for your response! I tried changing the settings to EY or CYS, but my nested run still stops at the same point. I am currently running it with the benchmark restart file.