[BUG/ISSUE] Possible parallelization or precision issue in PARANOX extension

[yantosca]

It seems that there is a parallelization or precision issue in the HEMCO PARANOX extension. When running with 5 and 8 cores we get very very small differences, which are enough to cause runs to be flagged as non-zero-diff.

Using configuration file hco58.yml
... Printing totals and differences
... Only showing variables with |absolute difference| > 0.0
Variable               Ref=5 cores              Dev=8 cores              Dev - Ref
EmisNO_Ship          : 1.5393503e-10          | 1.5393503e-10          | -2.144715586069626e-30 
InvPARANOX_NO        : 1.5393503e-10          | 1.5393503e-10          | -2.144715586069626e-30 
EmisNO_Total         : 1.9958073e-08          | 1.9958073e-08          | -2.144715586069626e-30 
```console

[yantosca]

The problem seems to be caused by NO and NO2 concentrations going into PARANOX being slightly different (starting with the 2nd chemistry timestep). When all operations except emissions are turned off, we get identical results.

I am trying to isolate which operation causes the differences. It could be deposition, given that we compute and store N deposition for the soil NOx emissions. Digging further.

[yantosca]

It appears that the issue is caused in transport. I put some debug print at 4x5 grid box (11,5,1) with all operations turned on and as you can see, the NO and NO2 start differing after the transport call at 00:20 into the run:

5 cores:

---> DATE: 2019/07/01  UTC: 00:20  X-HRS:      0.333333
 >>> NO  before tpcore    6.8647923275212405E-022
 >>> NO2 before tpcore    1.5925164678169723E-012
 >>> NO  after  tpcore    6.8254616900421252E-022
 >>> NO2 after  tpcore    1.5460629278609734E-012
 PARANOX ship emissions @          11           5
 NO             :    6.8254616900421252E-022
 NO2            :    1.5460629278609734E-012
 Emis [kg/m2/s] :    2.8198638221546904E-014
 SHIP_FNOx      :   0.73083455278538167     
 SHIP_DNOx      :    2.6053913825307973E-002
 SHIP_OPE       :   -6.7248229049146175     
 SHIP_MOE       :    1.2515978895066837E-004
 FNO_NOx        :    6.7684792889069566E-010
 FLUXNO         :    1.3948847043713018E-023

8 cores:

---> DATE: 2019/07/01  UTC: 00:20  X-HRS:      0.333333
 >>> NO  before tpcore    6.8647923275212405E-022
 >>> NO2 before tpcore    1.5925164678169723E-012
 >>> NO  after  tpcore    6.8254616900741325E-022
 >>> NO2 after  tpcore    1.5460629278610207E-012
 PARANOX ship emissions @          11           5
 NO             :    6.8254616900741325E-022
 NO2            :    1.5460629278610207E-012
 Emis [kg/m2/s] :    2.8198638221546904E-014
 SHIP_FNOx      :   0.73083455278538167     
 SHIP_DNOx      :    2.6053913825307973E-002
 SHIP_OPE       :   -6.7248229049146175     
 SHIP_MOE       :    1.2515978895066837E-004
 FNO_NOx        :    6.7684792889384888E-010
 FLUXNO         :    1.3948847043778002E-023

[yantosca]

I can confirm that this issue is not being caused by PARANOX per se, but it is an artifact of small differences that have been creeping in from GEOS-Chem operations.

In commit https://github.com/geoschem/geos-chem/commit/5bd68df14cf5e78717abbd687f6821bc6561a766, I added several fixes for numerical stability to GeosCore/tpcore_fvdas_mod.F90. Basically making sure that arrays are zeroed before they are used, as well as any arguments declared with INTENT(OUT).

I am looking into a similar issue in the VDIFF PBL mixing module. More to come.

[yantosca]

Several issues have been resolved in https://github.com/geoschem/HEMCO/pull/134 and https://github.com/geoschem/geos-chem/pull/1169. However, there are still some small differences that occur when transport, chemistry, and PBL mixing are turned on. I am currently attempting to isolate these differences and will post more at https://github.com/geoschem/geos-chem/issues/1157.

[yantosca]

We can close this issue because PR #134, which has been merged into the 13.4.0 development stream, solves this issue. Other outstanding issues with numerical precision/parallelization are not due to PARANOX and thus can be addressed in other Github issues (such as https://github.com/geoschem/geos-chem/issues/1157).

[yantosca]

Remaining parallelization issues in GEOS-Chem Classic simulations have been solved in PR https://github.com/geoschem/geos-chem/pull/1190.