geoschem / geos-chem

GEOS-Chem "Science Codebase" repository. Contains GEOS-Chem science routines, run directory generation scripts, and interface code. This repository is used as a submodule within the GCClassic and GCHP wrappers, as well as in other modeling contexts (external ESMs).
http://geos-chem.org
Other
163 stars 154 forks source link

[BUG/ISSUE] GEOS-FP Convection Change (June 2020-onward) can have large impacts on surface concentrations #1409

Open chfite opened 1 year ago

chfite commented 1 year ago

Description of the problem

First off, this message was also emailed to the Transport Working Group to get their input on the issue. However, I wanted to post this on here as well so that other users could be aware of the issue since it could potentially have large impacts on their results.

This is an issue we found with convection in the GEOS-FP files that begins in June 2020.

I had initially noticed in my GCClassic v13.3.4 nested simulation (0.25x0.3125, complexSOA_SVPOA) that my surface CO and PM2.5 concentrations began dropping off very drastically, WELL below observations, beginning around June 2020. Initially we assumed this could be a nested model or boundary condition issue but we saw the same drop-off in concentrations when using my 2x2.5 global simulation. So, we concluded the issue was likely in GEOS-FP.

I then ran a global transport tracer simulation at 2x2.5 resolution (non-local PBL turned on) to further diagnose the issue. Please see the attached PDF file with some figures of the model results. The COUniformEmis25dayTracer output was saved out as 7-day means for the 13 month period of August 2019 - August 2020.

Results are here: TransportTracerSim_Results.pdf

The budget diagnostic time series of CO confirms a sudden increased loss in CO (in the PBL) via convection, which also drives changes in the transport and mixing+emissions terms.

The plots showing the upward moist convective mass flux (CMFMC) show a near doubling that begins on June 1st, 2020. This can be seen in the monthly mean maps, as well as the global time series. For the CMFMC plots I used the raw 2x25 GEOS-FP that are 3-hourly. I have not explored CMFMC past August 2020, so I am not sure whether this issue gets resolved further on, but I suspect it doesn’t.

While looking at the GMAO website (https://gmao.gsfc.nasa.gov/GMAO_products/NRT_products.php#), we suspect that the convection update that occurred on March 30, 2020 (updated on April 7th, 2020) could maybe be the culprit, although that does not exactly correspond with when we see the doubling in CMFMC on June 1st.

We want users to be aware of this issue with GEOS-FP since it can have large impacts on the convection, mixing, and wet deposition terms for species.

yantosca commented 1 year ago

Adding some discussion to this thread:

@ltmurray wrote:

Whereas GMAO updated convection at the end of March, they performed a few months overlap with the old version and the new version. The GCSC decided we would provide the old stream until it was no longer available in order to avoid having such large transport discontinuities in the middle of the COVID shutdown period for anyone studying air quality during that time. That would explain the delay you see.

@aschuh wrote:

Unless I'm missing something, this has been a known change in FP for a long time, right? Going from RAS to GrellFreitas convection. I made plots of the conv mass flux diffs months ago and made several comments to folks on it, as I was drafting a paper on GC convection, submitted to acpd maybe a month ago. This is why most folks would be safer using a reanalysis. In fact, this is why reanalysis is created. For some reason most people (in the CO2 world at least) use FP despite these risks, I guess assuming 0.25deg must be viewed as better than 0.5deg for some reason (?) or FP has more data being assimilated (?). I've explained the exact risk associated with this transition many times. Honestly, happier to see it pop up in this context then as an inaccurate inference in a nature paper : ). Again, this isn't a bug, its exactly what is to be expected in the forward stream of a real time DA system, unless I'm missing something.

I made initial comparisons over 3-4 months early in the transition (shown in the figure attached) and discussed with Hongyu I believe. I was hoping to get a full year of the new FP and then run that year climatologically against a year of the old. Maybe 4-5 years, that should give good indication of expected diffs, at least for long lived tracers. Can't speak for short lived tracers but I'd be shocked if it didn't have a big effect on CO2 meridional gradients.

I've attached a quick and dirty image I made of the diffs back in May in the doc. This is zonally averaged CMFMC, i.e. a curtain, vertical by latitude. Forget the units, but they are the same across panels. Sorry, just indices, south pole to left, north pole to right, vertical model levels in y-axis. Top panel is RAS, middle is GF and bottom is difference. 3-4 months in avg, don't recall which. As was mentioned, huge changes. Plus there is this little pocket of zero conv mass flux around the 20th model level you'll notice, that was a concern as well since I'm not positive how tracers will move across it.

GF_vs_RAS.docx

As far I understand it, there really isn't much to do here, just wait for the next full reanalysis run? Feel free to correct me if I'm seeing this wrong.

Andrea Molod wrote:

i am going to make a comment about this that is from memory.....when the switch to the FP system with the new model took place in 2020, a few months in to the run the issue with the (incorrect) mass flux was detected. this is not a ras-gf change only, it was an outright bug. there is of course a change from RAS to GF here (the GF ones should be larger as the “scale aware” “squelching” does not kick in for GF until higher resolution), but this was a bug that was discovered.

now here is where it is.... foggy. i thought they ended up doing a re-wind with the correct output and re-ran that first few month time period, and replaced the FP on our servers. if the output was grabbed before that replacement happened, this could be what you are seeing.

if they decided not to do a re-run but just “live with the bug” and fix it going forward, there may be a short time period with bad fluxes that ends at some point and then the fluxes look ok.

the best person to ask about all this and get the details (dates, etc..) from is rob lucchessi.

also, here is the web site that you can click around in to see more info: https://gmao.gsfc.nasa.gov/GMAO_products/NRT_products.php#

one last “point” here – if the re-run was NOT done, it is also not a big deal to request that it be done. we have a mechanism called “exact replay” that minimizes the cost of the re-run. either christoph or emma knowland can make this happen if the request comes from the GC community....

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the Stale bot from closing this issue.

cdholmes commented 1 month ago

According to the timeline below, none of the GEOS-FP data contained in the GEOS-Chem archive should be affected by the convection bug mentioned by Andrea Molod. The issue still remains that GEOS-Chem Classic running with GEOS-FP meteorology after 1 June 2020 produces unrealistically low surface concentrations of CO, among other things.

Timeline based on GMAO release notes.

cdholmes commented 1 month ago

If any other GMAO products (e.g. GEOS-CF, GEOS-IT) are based on GEOS-5.25.1 and later they likely also produce unrealistic surface CO concentrations when used in GEOS-Chem Classic.

lizziel commented 1 week ago

After discussion with Randall and Daniel, we have decided to do 1-year transport tracer benchmarks for 2021 comparing MERRA2, GEOS-FP, and GEOS-IT, for both GCHP and GC-Classic. I will follow up here with results when they are available.

aschuh commented 1 week ago

STRONGLY recommend, I have not had the time to do this myself but I feel it is pretty critical.

I believe 1 year is probably sufficient w/ passive tracers (I assume?).    With 2-3 years, you really see the seasonal patterns emerge clearly but 1 will give you great info.  What tracers will be run?  For tracers tied to NH emissions (CO2, fossil/anthro, etc), it is sort of important to see these effects.  Will you have some decay tracers, full passive, a few zonal, etc?  Not sure what the full suite looks like I guess.

andrew

lizziel commented 1 week ago

@aschuh, the benchmarks will include the full set of transport tracers. Plots/tables generated will be everything typically included in the transport tracer benchnmark. The restart file will be the same one used for the MERRA2 2019 benchmariks.

The full set of transport tracers is listed on the wiki here. You can see what types of plots and tables will be generated from the transport tracer benchmark section of the 14.2.0 wiki page here.