[BUG/ISSUE] Picking up Emission Data in Nested Simulations

[jdsouz]

Report a GEOS-Chem bug or technical issue

Describe the bug:

I ran a GEOS-Chem nested simulation, that ran for five days and successfully completed. However, the data I received was not ideal. It appeared that around January 6 or 7, the simulation stopped reading emissions.

Expected behavior:

I expected to see a simulation with accurate representations of emission concentrations.

Actual behavior:

Black Carbon and PM2.5 produced completely normal results, but aerosols and ozone, among other variables, became undetected around January 6 and 7. I have attached Panoply animations of the various emission simulations below. As you can see the boundary conditions were normal, but the main nested run failed to produce emissions.

Steps to reproduce: the bug:

1. load_if17
2. make real clean
3. make -j4 build

Run commands

1. vi CopyRunDirs.input (edited run dates to 1/1/2018 to 6/1/2018, chose standard merra2_4x5)
2. ./gcCopyRunDirs
3. cd ~/GC/rundirs/merra2_4x5_standard/
4. vi HISTORY.rc (changed frequency and duration of Aerosol, AerosolMass, and SpeciesConc. Also set boundary conditions)
5. cp (other run directory)/a_run_geos_chem ~/GC/rundirs/merra2_4x5_standard

For the nested grid run, I went into HEMCO_Config.rc and:

6. Set GC_BCs to true

7. Changed directory to "BC (run directory)/GEOSChem.BoundaryConditions.$YYYY$MM$DD$HH$MNz.nc4 SpeciesBC"

8. Changed input.geos to boundary conditions dimensions.

9. Followed additional suggestions found here: https://github.com/geoschem/geos-chem/issues/401.

Error messages

No error messages.

Required information:

Your GEOS-Chem version and runtime environment:

• GEOS-Chem version: 12.8.2
• Compiler version: __
• netCDF version: __
• netCDF-Fortran version (if applicable): __

• Did you run on a computational cluster, on the AWS cloud: __

  • If you ran on the AWS cloud, please specify the Amazon Machine Image (AMI) ID: __
• Are you using GEOS-Chem "out of the box" (i.e. unmodified): __
  • If you have modified GEOS-Chem, please list what was changed: __

Input and log files to attach

• lastbuild: __
• input.geos: __
• HEMCO_Config.rc: __
• GEOS-Chem "Classic" log file: __
• HEMCO.log: __
• slurm.out or any other error messages from your scheduler: File is too big.
• Any other error messages: __

Additional context

Attached are the Panoply simulations that show the issue:

Expected Simulation Result:

Simulations that did not turn out as expected:

[yantosca]

@jdsouz thanks for writing. Can you attach the log files from the simulation, as well as the input files (input.geos and HEMCO_Config.rc)?

[jdsouz]

Yes, I can. They are attached here below.

input.geos.txt HEMCO_Config.rc.txt

The log file is too big to upload on Github, unfortunately.

[yantosca]

Thanks for sending the files. I think it may be because certain offline emissions are only available up to 2017, but your run goes into 2018. See:

#==============================================================================
# --- Offline biogenic VOC emissions ---
#==============================================================================
(((OFFLINE_BIOGENICVOC
0 BIOGENIC_ACET      $ROOT/OFFLINE_BIOVOC/v2019-10/0.5x0.625/$YYYY/$MM/biovoc_05.$YYYY$MM$DD.nc ACET_MEGAN    1980-2017/1-12/1-31/* C xy kgC/m2/s ACET -   4 2
... etc ...
#==============================================================================
# --- Offline sea salt emissions ---
#==============================================================================
(((OFFLINE_SEASALT
(((.not.SeaSalt
0 SEASALT_SALA    $ROOT/OFFLINE_SEASALT/v2019-01/0.5x0.625/$YYYY/$MM/seasalt_05.$YYYY$MM$DD.nc SALA_TOTAL   1980-2017/1-12/1-31/* C xy kg/m2/s SALA   - 3 2
0 SEASALT_SALC   
... etc ...

You should consider using the online emissions for years where the offline emissions are not available.

[jdsouz]

Hi @yantosca, thank you for your suggestion. I tried changing the offline emissions to online, and ran again for a shorter time period (1/1 to 1/14), but I still got the same error where emissions went away around January 6/7.

I do not think that this is the issue because the boundary conditions run successfully picked up all the emissions from January 1 all the way to May 31. Do you have any other suggestions as to what might have gone wrong? Thanks!

[jdsouz]

Hi, I just wanted to follow up and ask if anyone on the support team had an idea of what went wrong here. Thanks!

[jdsouz]

Hi all, just following up on this query. At your earliest convenience, please let me know if you see anything that could fix my emission gathering problem.

[yantosca]

I think I see the problem. Your HEMCO_Config.rc file has the latest year of offline dust emissions as 2017.

0 EMIS_DST1 . . . 1980-2017/1-12/1-31/\* C xy kg/m2/s DST1 606 3 2
0 EMIS_DST2 . . . 1980-2017/1-12/1-31/\* C xy kg/m2/s DST2 606 3 2
0 EMIS_DST3 . . . 1980-2017/1-12/1-31/\* C xy kg/m2/s DST3 606 3 2
0 EMIS_DST4 . . . 1980-2017/1-12/1-31/\* C xy kg/m2/s DST4 606 3 2

and also for the offline seasalt:

0 SEASALT_SALA  . . . 1980-2017/1-12/1-31/\* C xy kg/m2/s SALA   - 3 2
0 SEASALT_SALC  . . . 1980-2017/1-12/1-31/\* C xy kg/m2/s SALC   - 3 2

You would need to update the last year of those emissions to 2019.

What happened was that the extra years of emissions were added after the version you are using was released.

[jdsouz]

Hi @yantosca , I took your advice and updated the offline emissions to 2019. However, I am still running into the same problem. I do not think it is an issue of emissions accounting, but rather, a bug within the nested simulation itself. I claim this because the boundary conditions run worked flawlessly from Jan 2018 to Jun 2018. Is there anything that happens in the January 5-8 timeframe that could cause emissions not to show up in my nested simulation, but work fine in my boundary sim?

I know you may not have a clear answer on this right now, so please let me know if I can help you in any way. Thank you again for your continued help in this matter.

[yantosca]

Have you tried to debug the simulation? You can compile with the debug flags on from the run directory:

mkdir debug
cd debug
cmake ../CodeDir -DCMAKE_BUILD_TYPE=Debug
make -j install
cd ..

Then run the simulation. That might show any further errors.

We are prioritizing the 13.0.0 release now so there may be some delays between replies.

[jdsouz]

Hi @yantosca , I ran the simulation again with debug flags on. I shortened the simulation to 10 days because I did not want to take up excess memory or storage on the server. I used the same HEMCO file that I used last time, and I have pictures of portions of the slurm file below.

Notably, on January 6, starting at 3:00 am, it appears in the slurm file that there is an issue picking up NOx and N20 emissions. On line 66431643176666490104 of the file, it says:

### NOx direct delta: NaN kg This is followed up on the next line by ### N2O direct delta: NaN kg

The rest of the file is flooded with messages similar to this: Infinity in DO_CLOUD_CONVECTION! K, IC, Q(K,IC): 8 154 NaN It starts out with only a few of these messages at 3:10 am on January 6, but more of these messages pop up at 3:20 am, with more and more messages popping up later on that day and throughout the rest of the 10-day simulation. I have

This is the NaN error that pops up at 3 am on January 6:

This is the second error that comes up throughout the file (shown for 3:20 am):

Can you please let me know what these messages mean and how to potentially fix them? The slurm file is too big to upload on github, but please let me know if you need to see any other parts of the file, and I will send screenshots.

Thank you in advance for your help.

[yantosca]

Hi @jdsouz. I believe this error is the same as in https://github.com/geoschem/geos-chem/issues/402. Apparently, there is a div-by-zero happpening in UCX when the concentrations of either NOx or O3 get too low.

You can add the same fix as in commit e4a632bf4d5f8790797c8b6b3f84d883cf867261 to your code and see if that fixes the problem. Basically we don't do the division if the denominator is zero for a few expressions.

[yantosca]

Hi @jdsouz... just checking in if you were able to fix your issue by following the fix noted above. If so, we can close this issue out.

[jdsouz]

Hi @yantosca , the solution above compounded with changing the UCXNETCDF to False allowed the nested simulation to run. However, after the run got around 75% of the way through, I saw an error in the slurm file, but nothing out of the ordinary on the HEMCO.log file. I have attached a screenshot of the error below (the slurm file was too big to upload to Github). There seems to be a netCDF error. Please let me know where you think this is originating from! Thank you!

[jdsouz]

Hi @yantosca , I was just wondering if you had a chance to look at the error message above. I know your team is super busy with the new GEOS Chem release, so I completely understand the delay. I look forward to your response!

[msulprizio]

@jdsouz It's not clear what's causing the error from the snipped you provided. Here are some things to try to diagnose the issue:

1. If you are still running your simulation with debug flags on, try turning them off (i.e. using cmake ../CodeDir -DCMAKE_BUILD_TYPE=Release). It may be that running your simulation for a long simulation with debug flags turned on is causing issues. We recommend using the debug flags for short (1 day or less) simulations if possible, then once you're done debugging turn them off for your production run.

2. If you're still running the 10-day simulation, try running a shorter (e.g. 1-day) simulation to see if your run crashes at the same spot.

[yantosca]

I will close out this issue for now. Please feel free to submit a new issue if you are still having problems.