search

GEOS-ESM / GEOSgcm_GridComp

Repository containing the physics and IAU code for the GEOS Earth System Model
Apache License 2.0
9 stars 7 forks source link

SCM Failures due to RRTMG Update #399

Open mathomp4 opened 3 years ago

mathomp4 commented 3 years ago

So, this is a fun one and I'm not sure how to solve it. In the latest version of GEOSgcm_GridComp we incorporated changes from @dr0cloud that detects negative inputs into RRTMG (#393). And this is a Good Thing™ as negative inputs to RRTMG are usually not wanted.

But, it turns out, some of the SCM cases seem to have negative inputs as last night's tests showed:

In each case, it's cfc12vmr that triggers it. It's possible there are other negative inputs, but cfc12vmr is the first to trigger, and the Fortran stops.

I tried to look around GEOS and I'm sort of certain CFC12 is coming from the pchem.species.CMIP-5.MERRA2OX.197902-201706.z_91x72.nc4 file. I took a look and I'm pretty sure it's never negative in the file.

Now the question is: who to ask for help...because who is causing it to be negative?

I'll mention @gmao-esherman and @amolod as they are the SCM gurus I know of. And maybe @mmanyin as an overall knower of Chemistry knowledge.

Perhaps, though, this is due to GF/Moist so maybe @srfreita or @narnold1 or @wmputman ? I do see this in the log file:

 GF is doing transp and wet removal of: CFC12

so perhaps GF did something weird?

srfreita commented 3 years ago

Here is your simulation? I want to take a look at the log files. Saulo

From: Matthew Thompson notifications@github.com Reply-To: GEOS-ESM/GEOSgcm_GridComp reply@reply.github.com Date: Wednesday, January 13, 2021 at 7:48 AM To: GEOS-ESM/GEOSgcm_GridComp GEOSgcm_GridComp@noreply.github.com Cc: "Freitas, Saulo R. (GSFC-6100)[UNIVERSITIES SPACE RESEARCH ASSOCIATION]" saulo.r.freitas@nasa.gov, Mention mention@noreply.github.com Subject: [EXTERNAL] [GEOS-ESM/GEOSgcm_GridComp] SCM Failures due to RRTMG Update (#399)

So, this is a fun one and I'm not sure how to solve it. In the latest version of GEOSgcm_GridComp we incorporated changes from @dr0cloudhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdr0cloud&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395185337%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SV%2BS3H1GAk%2FU81cEe2yHAFr87jtcFN0BknPkTATT8Nk%3D&reserved=0 that detects negative inputs into RRTMG (#393https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGEOS-ESM%2FGEOSgcm_GridComp%2Fpull%2F393&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395195296%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ztp1Nz7LTaqAnWHeU2ofNphwlER%2BNxhUYk1Yj8wPds4%3D&reserved=0). And this is a Good Thing™ as negative inputs to RRTMG are usually not wanted.

But, it turns out, some of the SCM cases seem to have negative inputs as last night's tests showed: · TRMM_LBA

file:rrtmg_lw_rad.F90, line: 378

minval(cfc12vmr): -8.1840096E-22

negative values in input: cfc12vmr

· arm_97jul

file:rrtmg_lw_rad.F90, line: 378

minval(cfc12vmr): -1.5127862E-20

negative values in input: cfc12vmr

· merra_arm97jul

file:rrtmg_lw_rad.F90, line: 378

minval(cfc12vmr): -1.9174425E-20

negative values in input: cfc12vmr

· merra_armsgp

file:rrtmg_lw_rad.F90, line: 378

minval(cfc12vmr): -1.0579275E-20

negative values in input: cfc12vmr

· merra_namecore

file:rrtmg_lw_rad.F90, line: 378

minval(cfc12vmr): -7.2872787E-23

negative values in input: cfc12vmr

· merra_trmm

file:rrtmg_lw_rad.F90, line: 378

minval(cfc12vmr): -5.3364247E-22

negative values in input: cfc12vmr

· merra_twp

file:rrtmg_lw_rad.F90, line: 378

minval(cfc12vmr): -4.6316816E-23

negative values in input: cfc12vmr

In each case, it's cfc12vmr that triggers it. It's possible there are other negative inputs, but cfc12vmr is the first to trigger, and the Fortran stops.

I tried to look around GEOS and I'm sort of certain CFC12 is coming from the pchem.species.CMIP-5.MERRA2OX.197902-201706.z_91x72.nc4 file. I took a look and I'm pretty sure it's never negative in the file.

Now the question is: who to ask for help...because who is causing it to be negative?

I'll mention @gmao-eshermanhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgmao-esherman&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395195296%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wQJipdNXDM%2BpEVour7%2B0MkzRQAutvO7bm4QH6IelMLk%3D&reserved=0 and @amolodhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Famolod&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395205249%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MpDfO51gYUuIRHLxpjJ8Jsd7xdGGxiE0JI6DjP7wQLw%3D&reserved=0 as they are the SCM gurus I know of. And maybe @mmanyinhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmmanyin&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395205249%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YYDjeue3Rs42%2B9Jp16IlQt%2FS9C%2FSOEUzN6aUy5hJWaQ%3D&reserved=0 as an overall knower of Chemistry knowledge.

Perhaps, though, this is due to GF/Moist so maybe @srfreitahttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsrfreita&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395205249%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wHjAfNsUW5XET7zODm40nr9NccbBRlJWh609I0CElUE%3D&reserved=0 or @narnold1https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnarnold1&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395215208%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nojh2JeR8JwEMWAH7jYIHjWZAWXqrGQaC8hDIMx3OLM%3D&reserved=0 or @wmputmanhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwmputman&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395215208%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uRbWpHyTCNXE2Kxoxaq9A%2B1F5SJQdmTvAxO2nfWFv44%3D&reserved=0 ? I do see this in the log file:

GF is doing transp and wet removal of: CFC12

so perhaps GF did something weird?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGEOS-ESM%2FGEOSgcm_GridComp%2Fissues%2F399&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395215208%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ieNF4FGkLw8%2FXfFeTLTUfFtTVrck%2FGdKCjZ9KdIRc%2Bs%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FANAREE6U54GSBVQAI3HYFH3SZWJDHANCNFSM4WAYFBCA&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7C7d485dfab68a4ac880ab08d8b7c197bc%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461389395225169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GtpHZF92WdSmiArXltLhG1Iy9JyVkhjTqglwo9JNQA4%3D&reserved=0.

mathomp4 commented 3 years ago

@srfreita You can find an example nightly run here:

/discover/nobackup/mathomp4/SystemTests/runs/SCM/CURRENT/run/arm_97jul
mathomp4 commented 3 years ago

Also, a note. I decided to run (in a separate location) with:

USE_TRACER_TRANSP: 0

because...well, I don't know what I'm doing. Plus side, it changed the error; minus side, it made it more negative.

With USE_TRACER_TRANSP: 1:

 minval(cfc12vmr): -1.5127862E-20

with USE_TRACER_TRANSP: 0:

 minval(cfc12vmr): -3.0200277E-20
srfreita commented 3 years ago

Test also with USE_TRACER_TRANSP_UW: 0

From: Matthew Thompson notifications@github.com Reply-To: GEOS-ESM/GEOSgcm_GridComp reply@reply.github.com Date: Wednesday, January 13, 2021 at 8:26 AM To: GEOS-ESM/GEOSgcm_GridComp GEOSgcm_GridComp@noreply.github.com Cc: "Freitas, Saulo R. (GSFC-6100)[UNIVERSITIES SPACE RESEARCH ASSOCIATION]" saulo.r.freitas@nasa.gov, Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [GEOS-ESM/GEOSgcm_GridComp] SCM Failures due to RRTMG Update (#399)

Also, a note. I decided to run (in a separate location) with:

USE_TRACER_TRANSP: 0

because...well, I don't know what I'm doing. Plus side, it changed the error; minus side, it made it more negative.

With USE_TRACER_TRANSP: 1:

minval(cfc12vmr): -1.5127862E-20

with USE_TRACER_TRANSP: 0:

minval(cfc12vmr): -3.0200277E-20

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGEOS-ESM%2FGEOSgcm_GridComp%2Fissues%2F399%23issuecomment-759446031&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7Cc1ebacc523bd4130c14e08d8b7c6b332%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461412074847281%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sHs6hlor%2BZktuIr9jtMJTmLYHdKARm4h%2FLSklunXLsE%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FANAREE3DGD6UD4ITAJAYN2TSZWNMJANCNFSM4WAYFBCA&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7Cc1ebacc523bd4130c14e08d8b7c6b332%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461412074847281%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=J%2FqI6DEsp6ZR3L36cI94pzUiMBzaqRlehv5HWSQ461s%3D&reserved=0.

mathomp4 commented 3 years ago

With USE_TRACER_TRANSP_UW: 0:

  minval(cfc12vmr): -1.5127862E-20

With USE_TRACER_TRANSP_UW: 0 and USE_TRACER_TRANSP: 0:

  minval(cfc12vmr): -3.0200277E-20

So it seems to have no effect.

amolod commented 3 years ago

hi matt,

not sure what is happening in the full model, but my impression is that SCM should run with fixed pchem constituents, not prod and loss. in which case the cfc’s are not friendly to anyone – dyn or moist or turb....

i need to dig a little to see what the proper settings are for this and what the proper pchem species file (or climatology) is.

i wonder if some of these settings in the AGCM morphed over time with the model defaults....

am i correct that this does not happen with this tag in the full model? does the model run with the same pchem settings (ie., use P and L with the same relaxation time)?

thanks

From: Saulo Freitas notifications@github.com Reply-To: GEOS-ESM/GEOSgcm_GridComp reply@reply.github.com Date: Wednesday, January 13, 2021 at 8:39 AM To: GEOS-ESM/GEOSgcm_GridComp GEOSgcm_GridComp@noreply.github.com Cc: "Molod, Andrea M. (GSFC-6101)" andrea.m.molod@nasa.gov, Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [GEOS-ESM/GEOSgcm_GridComp] SCM Failures due to RRTMG Update (#399)

Test also with USE_TRACER_TRANSP_UW: 0

From: Matthew Thompson notifications@github.com Reply-To: GEOS-ESM/GEOSgcm_GridComp reply@reply.github.com Date: Wednesday, January 13, 2021 at 8:26 AM To: GEOS-ESM/GEOSgcm_GridComp GEOSgcm_GridComp@noreply.github.com Cc: "Freitas, Saulo R. (GSFC-6100)[UNIVERSITIES SPACE RESEARCH ASSOCIATION]" saulo.r.freitas@nasa.gov, Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [GEOS-ESM/GEOSgcm_GridComp] SCM Failures due to RRTMG Update (#399)

Also, a note. I decided to run (in a separate location) with:

USE_TRACER_TRANSP: 0

because...well, I don't know what I'm doing. Plus side, it changed the error; minus side, it made it more negative.

With USE_TRACER_TRANSP: 1:

minval(cfc12vmr): -1.5127862E-20

with USE_TRACER_TRANSP: 0:

minval(cfc12vmr): -3.0200277E-20

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGEOS-ESM%2FGEOSgcm_GridComp%2Fissues%2F399%23issuecomment-759446031&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7Cc1ebacc523bd4130c14e08d8b7c6b332%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461412074847281%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sHs6hlor%2BZktuIr9jtMJTmLYHdKARm4h%2FLSklunXLsE%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FANAREE3DGD6UD4ITAJAYN2TSZWNMJANCNFSM4WAYFBCA&data=04%7C01%7Csaulo.r.freitas%40nasa.gov%7Cc1ebacc523bd4130c14e08d8b7c6b332%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461412074847281%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=J%2FqI6DEsp6ZR3L36cI94pzUiMBzaqRlehv5HWSQ461s%3D&reserved=0.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGEOS-ESM%2FGEOSgcm_GridComp%2Fissues%2F399%23issuecomment-759453879&data=04%7C01%7Candrea.m.molod%40nasa.gov%7Cfb21e50a32b14762f8ba08d8b7c89e9f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461419586900681%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UnFXXTeA76ud6jy40iAO2fw%2FHTG1j9sc5J7sntOblY8%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGAQKCPKORBGDUV4FLLQ3D3SZWO75ANCNFSM4WAYFBCA&data=04%7C01%7Candrea.m.molod%40nasa.gov%7Cfb21e50a32b14762f8ba08d8b7c89e9f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461419586910629%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tCnGTYekcWxotlf7%2BzLDZrK0rFwNPrY%2F%2FWsdLyKgUyQ%3D&reserved=0.

mathomp4 commented 3 years ago

not sure what is happening in the full model, but my impression is that SCM should run with fixed pchem constituents, not prod and loss. in which case the cfc’s are not friendly to anyone – dyn or moist or turb.... i need to dig a little to see what the proper settings are for this and what the proper pchem species file (or climatology) is. i wonder if some of these settings in the AGCM morphed over time with the model defaults.... am i correct that this does not happen with this tag in the full model? does the model run with the same pchem settings (ie., use P and L with the same relaxation time)?

@amolod I don't think it is running with prod-loss. SCM runs essentially like how the AGCM runs if you ask for climatological aerosols. So AGCM.rc has:

# AeroChem Environment
# --------------------
     OX_RELAXTIME: 259200.    # Standard Options:  GCM: 259200., DAS: 0.000
    CH4_RELAXTIME: 259200.
    N2O_RELAXTIME: 259200.
  CFC11_RELAXTIME: 259200.
  CFC12_RELAXTIME: 259200.
 HCFC22_RELAXTIME: 259200.
    H2O_RELAXTIME: 259200.
    OX_FRIENDLIES: ANALYSIS:DYNAMICS:TURBULENCE:MOIST
   CH4_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
   N2O_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
 CFC11_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
 CFC12_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
   H2O_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
HCFC22_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
       pchem_clim: species.data
       pchem_clim_years: 39

and we link species.data to:

pchem.species.CMIP-5.MERRA2OX.197902-201706.z_91x72.nc4

which is equivalent to the AGCM linkbcs:

# MERRA-2 Ozone Data (AGCM.rc:  pchem_clim_years = 39-Years)
# ----------------------------------------------------------
/bin/ln -sf /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds/Shared/pchem.species.CMIP-5.MERRA2OX.197902-201706.z_91x72.nc4 species.data
mathomp4 commented 3 years ago

I believe every SCM case is currently within the MERRA2OX time period. The error if we were running in, say, 2020 would be much different (I think @bena-nasa knows the error by heart).

amolod commented 3 years ago

nathan there is an option to just read this from a climatology file and NOT have them friendly to anyone. this is the configuration i don’t remember how to enforce. maybe mike manyin knows offhand.... maybe set relaxtime to 0? i cc’d mike here now....

otherwise the budgets need to be turned on to show who is driving it negative. it may even be the dry mass constraint tendency. but again, elliot added the option to get ALL the budget terms. i forgot the syntax....

From: Matthew Thompson notifications@github.com Reply-To: GEOS-ESM/GEOSgcm_GridComp reply@reply.github.com Date: Wednesday, January 13, 2021 at 9:37 AM To: GEOS-ESM/GEOSgcm_GridComp GEOSgcm_GridComp@noreply.github.com Cc: "Molod, Andrea M. (GSFC-6101)" andrea.m.molod@nasa.gov, Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [GEOS-ESM/GEOSgcm_GridComp] SCM Failures due to RRTMG Update (#399)

not sure what is happening in the full model, but my impression is that SCM should run with fixed pchem constituents, not prod and loss. in which case the cfc’s are not friendly to anyone – dyn or moist or turb.... i need to dig a little to see what the proper settings are for this and what the proper pchem species file (or climatology) is. i wonder if some of these settings in the AGCM morphed over time with the model defaults.... am i correct that this does not happen with this tag in the full model? does the model run with the same pchem settings (ie., use P and L with the same relaxation time)?

@narnold1https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnarnold1&data=04%7C01%7Candrea.m.molod%40nasa.gov%7C28cec9ad486c47f7361d08d8b7d0cf36%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461454743117310%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Ly3XxCDub5Dn7Hb4m6JAEYavlPxRn27IrEBRy4Wki7A%3D&reserved=0 I don't think it is running with prod-loss. SCM runs essentially like how the AGCM runs if you ask for climatological aerosols. So AGCM.rc has:

AeroChem Environment

--------------------

 OX_RELAXTIME: 259200.    # Standard Options:  GCM: 259200., DAS: 0.000

CH4_RELAXTIME: 259200.

N2O_RELAXTIME: 259200.

CFC11_RELAXTIME: 259200.

CFC12_RELAXTIME: 259200.

HCFC22_RELAXTIME: 259200.

H2O_RELAXTIME: 259200.

OX_FRIENDLIES: ANALYSIS:DYNAMICS:TURBULENCE:MOIST

CH4_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST

N2O_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST

CFC11_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST

CFC12_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST

H2O_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST

HCFC22_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST

   pchem_clim: species.data

   pchem_clim_years: 39

and we link species.data to:

pchem.species.CMIP-5.MERRA2OX.197902-201706.z_91x72.nc4

which is equivalent to the AGCM linkbcs:

MERRA-2 Ozone Data (AGCM.rc: pchem_clim_years = 39-Years)

----------------------------------------------------------

/bin/ln -sf /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds/Shared/pchem.species.CMIP-5.MERRA2OX.197902-201706.z_91x72.nc4 species.data

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGEOS-ESM%2FGEOSgcm_GridComp%2Fissues%2F399%23issuecomment-759489970&data=04%7C01%7Candrea.m.molod%40nasa.gov%7C28cec9ad486c47f7361d08d8b7d0cf36%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461454743117310%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VUSV8xLFvo57jMpzda%2BegGldMZr80EpW3mJoR35XDv8%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGAQKCLSKZKV6GVNGTI7TQLSZWV3XANCNFSM4WAYFBCA&data=04%7C01%7Candrea.m.molod%40nasa.gov%7C28cec9ad486c47f7361d08d8b7d0cf36%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461454743127275%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=02HPGExWsLntqWAaESAySdckkdylbdf9MZEykoaiPgM%3D&reserved=0.

mmanyin commented 3 years ago

In order to make a PCHEM species UN-FRIENDLY, just comment out of the AGCM.rc entry

_FRIENDLIES . As for the P&L, the comments say that you can fix a species to a climatology by using a very short relaxation time. I would guess that something of a minute or less would qualify as a very short time, e.g. CFC11_RELAXTIME: 60. @gmao-esherman would be the best to ask about the Tendency diagnostics, relative to PCHEM.
amolod commented 3 years ago

hi, so to make it work we can turn off the friendlies and set the short relaxation time. but it would still be interesting to find out which tendency term is making the negatives.... perhaps we can talk with elliot.... my two cents.

nathan? does this make sense?

From: Mike Manyin notifications@github.com Reply-To: GEOS-ESM/GEOSgcm_GridComp reply@reply.github.com Date: Wednesday, January 13, 2021 at 11:23 AM To: GEOS-ESM/GEOSgcm_GridComp GEOSgcm_GridComp@noreply.github.com Cc: "Molod, Andrea M. (GSFC-6101)" andrea.m.molod@nasa.gov, Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [GEOS-ESM/GEOSgcm_GridComp] SCM Failures due to RRTMG Update (#399)

In order to make a PCHEM species UN-FRIENDLY, just comment out of the AGCM.rc entry

_FRIENDLIES . As for the P&L, the comments say that you can fix a species to a climatology by using a very short relaxation time. I would guess that something of a minute or less would qualify as a very short time, e.g. CFC11_RELAXTIME: 60. @gmao-esherman would be the best to ask about the Tendency diagnostics, relative to PCHEM. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
gmao-esherman commented 3 years ago

I don't think the tendency diagnostic code will work with the SCM. The tendency diagnostic code depends on amending files in the experiment_dir/scratch directory by running bundleParser.py, which we don't have with SCM experiments. I would have to dedicate time to get the tendency diagnostic code to work with the SCM, which I can do if we need.

mathomp4 commented 3 years ago

As a note, I decided to print out CFC12_R before it goes into RRTMG_LW and it is:

 MATMAT CFC12_R:   3.6584221E-12  3.6581853E-12  3.6578904E-12  3.6575873E-12
  3.6572837E-12  3.6570222E-12  3.6567676E-12  3.6565087E-12  3.6562901E-12
  3.6560607E-12  3.6558907E-12  3.6557359E-12  3.6555628E-12  3.6553594E-12
  3.6551265E-12  3.6548863E-12  3.6546408E-12  3.6543893E-12  3.6540896E-12
  3.6537552E-12  3.6534573E-12  3.6532301E-12  3.6531151E-12  3.6531056E-12
  3.6531659E-12  3.6532596E-12  3.6532687E-12  3.6529395E-12  3.6517785E-12
  3.6489509E-12  3.6430108E-12  3.6290901E-12  3.5975578E-12  3.5506327E-12
  3.4880235E-12  3.3497510E-12  3.1168438E-12  2.7898267E-12  2.4500007E-12
  2.1643705E-12  1.8919935E-12  1.6283553E-12  1.4130674E-12  1.2089398E-12
  1.0112683E-12  8.1747646E-13  6.2055997E-13  4.3404969E-13  3.0619078E-13
  1.9660990E-13  1.2830440E-13  7.7564135E-14  4.6092914E-14  1.9308634E-14
  1.1711126E-14  6.2087104E-15  4.3828907E-15  2.6073413E-15  1.4312273E-15
  1.0326099E-15  7.5670181E-16  5.3495610E-16  3.4596812E-16  2.2476207E-16
  1.4249683E-16  9.3435165E-17  4.6579142E-17  2.1750064E-17  8.4597258E-18
  3.0211699E-18  6.4517606E-19 -1.5127862E-20

I think this says the negative value is at the top-of-atmosphere (it's flipped from GEOS normality).

narnold1 commented 3 years ago

Making species UN-FRIENDLY and changing the relaxtime as Mike suggests makes sense to me, as a fix for the SCM. But it sounds like we also need to verify this isn't triggered in the full model?

mathomp4 commented 3 years ago

Maybe somehow the extrapolation is getting a negative value (if it's, say, cubic ?). But then you'd figure the full model would go nuts much faster running on thousands of columns...

mathomp4 commented 3 years ago

Making species UN-FRIENDLY and changing the relaxtime as Mike suggests makes sense to me, as a fix for the SCM. But it sounds like we also need to verify this isn't triggered in the full model?

I can run a test if you like if you teach me what to set in AGCM.rc.

As for the full model, my GOCART.data run worked last night which is c24, so that means 3456 columns (times however many steps in a day) didn't trigger the RRTMG protections. I can try a higher-resolution GOCART.data run, I suppose.

amolod commented 3 years ago

matt, here is the recommendation from mike manyin:

In order to make a PCHEM species UN-FRIENDLY, just comment out of the AGCM.rc entry

_FRIENDLIES . As for the P&L, the comments say that you can fix a species to a climatology by using a very short relaxation time. I would guess that something of a minute or less would qualify as a very short time, e.g. CFC11_RELAXTIME: 60. you may want to do this with all of them... or with just cfc11, 12. if you are right about the interpolation it may just be cfc11, 12.... thanks! From: Matthew Thompson Reply-To: GEOS-ESM/GEOSgcm_GridComp Date: Wednesday, January 13, 2021 at 1:06 PM To: GEOS-ESM/GEOSgcm_GridComp Cc: "Molod, Andrea M. (GSFC-6101)" , Mention Subject: [EXTERNAL] Re: [GEOS-ESM/GEOSgcm_GridComp] SCM Failures due to RRTMG Update (#399) Making species UN-FRIENDLY and changing the relaxtime as Mike suggests makes sense to me, as a fix for the SCM. But it sounds like we also need to verify this isn't triggered in the full model? I can run a test if you like if you teach me what to set in AGCM.rc. As for the full model, my GOCART.data run worked last night which is c24, so that means 3456 columns didn't trigger the RRTMG protections. I can try a higher-resolution GOCART.data run, I suppose. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
mathomp4 commented 3 years ago

Hmm. Well the SCM did not like this:

# AeroChem Environment
# --------------------
     OX_RELAXTIME: 60.    # Standard Options:  GCM: 259200., DAS: 0.000
    CH4_RELAXTIME: 60.
    N2O_RELAXTIME: 60.
  CFC11_RELAXTIME: 60.
  CFC12_RELAXTIME: 60.
 HCFC22_RELAXTIME: 60.
    H2O_RELAXTIME: 60.
#    OX_FRIENDLIES: ANALYSIS:DYNAMICS:TURBULENCE:MOIST
#   CH4_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
#   N2O_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
# CFC11_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
# CFC12_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
#   H2O_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
#HCFC22_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
       pchem_clim: species.data
       pchem_clim_years: 39

It dies FAST with:

 Real*4 Resource Parameter: SYNCTQ:1.000000
 ERROR: cannot find export CH4
pe=00000 FAIL at line=06579    MAPL_Generic.F90                         <unknown error>
pe=00000 FAIL at line=00607    MAPL_Generic.F90                         <status=-1>
PHYSICS::SetServices                          1473
pe=00000 FAIL at line=00168    BaseProfiler.F90                         <Timer SetService likely does not find its pair>
pe=00000 FAIL at line=04565    MAPL_Generic.F90                         <status=1>
pe=00000 FAIL at line=04600    MAPL_Generic.F90                         <status=1>
AGCMSetServices                                690
pe=00000 FAIL at line=00168    BaseProfiler.F90                         <Timer AGCM likely does not find its pair>
pe=00000 FAIL at line=04567    MAPL_Generic.F90                         <status=1>
pe=00000 FAIL at line=04600    MAPL_Generic.F90                         <status=1>
GCMSetServices                                 232
pe=00000 FAIL at line=00168    BaseProfiler.F90                         <Timer GCM likely does not find its pair>
pe=00000 FAIL at line=04567    MAPL_Generic.F90                         <status=1>
pe=00000 FAIL at line=00529    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=00901    MAPL_CapGridComp.F90                     <status=1>
pe=00000 FAIL at line=00357    MAPL_Cap.F90                             <status=1>
pe=00000 FAIL at line=00214    MAPL_Cap.F90                             <status=1>
pe=00000 FAIL at line=00160    MAPL_Cap.F90                             <status=1>
pe=00000 FAIL at line=00134    MAPL_Cap.F90                             <status=1>
pe=00000 FAIL at line=00032    GEOSgcm.F90                              <status=1>
Abort(77771072) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 77771072) - process 0
mathomp4 commented 3 years ago

Realizing I might have gone too far, I just tried this in AGCM.rc:

# AeroChem Environment
# --------------------
     OX_RELAXTIME: 259200.    # Standard Options:  GCM: 259200., DAS: 0.000
    CH4_RELAXTIME: 259200.
    N2O_RELAXTIME: 259200.
  CFC11_RELAXTIME: 259200.
  CFC12_RELAXTIME: 60.
 HCFC22_RELAXTIME: 259200.
    H2O_RELAXTIME: 259200.
    OX_FRIENDLIES: ANALYSIS:DYNAMICS:TURBULENCE:MOIST
   CH4_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
   N2O_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
 CFC11_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
# CFC12_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
   H2O_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
HCFC22_FRIENDLIES: DYNAMICS:TURBULENCE:MOIST
       pchem_clim: species.data
       pchem_clim_years: 39

and I still got a crash:

 file:rrtmg_lw_rad.F90, line:         378
 minval(cfc12vmr): -4.3548214E-18
negative values in input: cfc12vmr
gmao-esherman commented 3 years ago

I have been trying to get the increment bundle code to work with the SCM. So far I have been unsuccessful. I will update if I get this working.

gmao-esherman commented 3 years ago

I was able to get the increment bundle code to work with the SCM. We just have to do what bundleParser.py would do by hand, which really isn't very hard. Please let me know if you would like to add some PCHEM species increments for your SCM experiment.

amolod commented 3 years ago

this is great elliot! i think adding them to even one of the experiments, just to see who is the culprit, would be enough to spot the issue. thanks

From: Elliot Sherman notifications@github.com Reply-To: GEOS-ESM/GEOSgcm_GridComp reply@reply.github.com Date: Wednesday, January 13, 2021 at 2:44 PM To: GEOS-ESM/GEOSgcm_GridComp GEOSgcm_GridComp@noreply.github.com Cc: "Molod, Andrea M. (GSFC-6101)" andrea.m.molod@nasa.gov, Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [GEOS-ESM/GEOSgcm_GridComp] SCM Failures due to RRTMG Update (#399)

I was able to get the increment bundle code to work with the SCM. We just have to do what bundleParser.py would do by hand, which really isn't very hard. Please let me know if you would like to add some PCHEM species increments for your SCM experiment.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGEOS-ESM%2FGEOSgcm_GridComp%2Fissues%2F399%23issuecomment-759694561&data=04%7C01%7Candrea.m.molod%40nasa.gov%7C8ddd73990d5749348efb08d8b7fba379%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461638707336831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=174eGP8bbN1ehFyJrgMpcVIU%2FbfHgSw6xLWjNMQQPsw%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGAQKCNZP6I2EN63AUAI6DTSZXZZLANCNFSM4WAYFBCA&data=04%7C01%7Candrea.m.molod%40nasa.gov%7C8ddd73990d5749348efb08d8b7fba379%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637461638707336831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rzuSZEcqIASyZTfe90yWSI%2BXOUOMrPrytbHJhSKR%2B3c%3D&reserved=0.

gmao-esherman commented 3 years ago

I ran the standard arm97 case for one time-step (15min), starting at the beginning most date (19970618 233000), and wrote out a netcdf file that contains the increments due to Chemistry, Turbulence, Moist, Dynamics, and H2O rescaling. The settings for the file collection are listed below.

geosscm_inc.template: '%y4%m2%d2%h2.nc4', geosscm_inc.archive: '%c/Y%y4/M%m2', geosscm_inc.format: 'CFIO', geosscm_inc.format: 'flat', geosscm_inc.mode: 'instantaneous', geosscm_inc.grid_label: SCMGRID, geosscm_inc.frequency: 001500, geosscm_inc.duration: 000000,

The following increments contained negative values: H2O rescaling, and Chemistry.

I guess the problem is in the H2O rescaling, or PCHEM, or both.

amolod commented 3 years ago

this is great elliot. can you tell me where this output lives?

From: Elliot Sherman notifications@github.com Reply-To: GEOS-ESM/GEOSgcm_GridComp reply@reply.github.com Date: Tuesday, January 19, 2021 at 2:26 PM To: GEOS-ESM/GEOSgcm_GridComp GEOSgcm_GridComp@noreply.github.com Cc: "Molod, Andrea M. (GSFC-6101)" andrea.m.molod@nasa.gov, Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [GEOS-ESM/GEOSgcm_GridComp] SCM Failures due to RRTMG Update (#399)

I ran the standard arm97 case for one time-step (15min), starting at the beginning most date (19970618 233000), and wrote out a netcdf file that contains the increments due to Chemistry, Turbulence, Moist, Dynamics, and H2O rescaling. The settings for the file collection are listed below.

geosscm_inc.template: '%y4%m2%d2%h2.nc4', geosscm_inc.archive: '%c/Y%y4/M%m2', geosscm_inc.format: 'CFIO', geosscm_inc.format: 'flat', geosscm_inc.mode: 'instantaneous', geosscm_inc.grid_label: SCMGRID, geosscm_inc.frequency: 001500, geosscm_inc.duration: 000000,

The following increments contained negative values: H2O rescaling, and Chemistry.

I guess the problem is in the H2O rescaling, or PCHEM, or both.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGEOS-ESM%2FGEOSgcm_GridComp%2Fissues%2F399%23issuecomment-763074532&data=04%7C01%7Candrea.m.molod%40nasa.gov%7C03f189b086b34b798ce108d8bcb01606%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637466811756275652%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=x447K9zjfUfGN4xWRyw704BB7iKh6VlCiN0NaxwbhH0%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGAQKCIWZ5IIWDFYHOBZZ63S2XME7ANCNFSM4WAYFBCA&data=04%7C01%7Candrea.m.molod%40nasa.gov%7C03f189b086b34b798ce108d8bcb01606%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637466811756444902%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZRjhoFAkY5XdL75D0%2F3iQ5YiGUnVnLb%2FGJa47EXJTbI%3D&reserved=0.

gmao-esherman commented 3 years ago

The experiment directory can be found here: /gpfsm/dnb32/esherman/arm97_pchemINC The netcdf files is in the experiment directory and called "arm_97jul.geosscm_inc.1997061823.nc4".

wmputman commented 3 years ago

@mmanyin , I'm running into a similar error in my c5760 (1.5km) GEOS run with just PChem (GOCART: None). I am seeing negative values in CH4 with the default timescales and friendlies. I can resolve this with: CH4_FIXED_VALUE: in AGCM.rc if I know what a good value is to choose? I need to complete just 12 more days of this run (2020-Feb 18-29th), would this be sufficient? CH4_FIXED_VALUE: 1875.0 (ppb) what are the units?

gmao-esherman commented 3 years ago

I just re-ran the SCM case I posted about previously (https://github.com/GEOS-ESM/GEOSgcm_GridComp/issues/399#issuecomment-763074532), but this time I commented out the H2O rescaling code in GEOS_PhysicsGridComp.F90. Everything else about the run is the same.

There is still a negative value for the cfc12vmr chemistry increment. All other increments are zero. This leads me to believe the negative value is originating in PCHEM, and likely not caused by H2O rescaling. I don't know why this is happening, or why it's happening now (why not in an Icarus tag?). I believe @mmanyin, Luke Oman, and Qing Liang are investigating this issue and working on a solution to prevent PCHEM from generating negative values.

mathomp4 commented 3 years ago

An update. @mmanyin has a proposed (non-zero-diff) fix for this issue in GEOS_PchemGridComp.F90 and I can say it passes arm_97jul now. I'm currently trying to remember how I can run the full suite of SCM cases to make sure they all work.

mathomp4 commented 3 years ago

Update: All the cases pass with the update from @mmanyin ! 🎉 🥳

mathomp4 commented 3 years ago

The update from @mmanyin is now in as https://github.com/GEOS-ESM/GEOSchem_GridComp/pull/116

As it is non-zero-diff, it might be a while to get in, but it is there.