Albedo's are bad at night with negative longitudes

Brief summary of bug

FSR and other reflected fields are bad at nighttime with negative longitudes.

General bug information

CTSM version you are using: ctsm1.0.dev009 Does this bug cause significantly incorrect results in the model's science? Yes

Configurations affected: both SP and BGC cases, any resolution

Details of bug

This shows up as an issue over the America's for both regional and global grids. For a simulation starting on Jan/1 at 0Z it is highest at hour 20Z. You see it in reflected fields such as FSR, but many other fields show it as well. And it happens even though both FSDS and COSZEN are correct. So the solar coming from cime/datm is correct, and shr_orb seems to be correct as well. As far as I can tell the determination of night and day ultimately comes from coszen, so I don't see how this is happening. I also don't see other dependences on longitude in the code that would explain this issue either.

This started out in issue #491. I'm putting it here so I can close that issue with the fix I have. This one will stay open until we can work on it.

As the first step to it, I am putting in an abort when a negative longitude is read in from the surface dataset.

Important details of your setup / configuration so we can reproduce the bug

SMS_D_Ld3.f09_g16.I2000Clm50SpGs.cheyenne_intel.clm-default

Add these settings:

shell_commands: ./xmlchange ATM_DOMAIN_FILE=domain.lnd.fv0.9x1.25_gx1v6.090309.negative.longitude.nc ./xmlchange ATM_DOMAIN_PATH=/glade/scratch/swensosc/sfcdata ./xmlchange LND_DOMAIN_FILE=domain.lnd.fv0.9x1.25_gx1v6.090309.negative.longitude.nc ./xmlchange LND_DOMAIN_PATH=/glade/scratch/swensosc/sfcdata/

user_nl_clm: fsurdat = '/glade/scratch/swensosc/sfcdata/surfdata_0.9x1.25_16pfts_Irrig_CMIP6_simyr2000_c170824.negative.longitude.nc'

It fails the BASELINE comparision with many large differences in a many fields see below. I expect that the differences would only be order of roundoff, since that's what we see in COSZEN and FSDS.

Important output or errors that show the problem

RMS lon                              2.5367E+02
 RMS DSL                              2.7679E+00
 RMS EFLX_GRND_LAKE                   6.6264E+01
 RMS EFLX_LH_TOT                      1.4267E+01
 RMS EFLX_LH_TOT_R                    1.6778E+01
 RMS FCEV                             1.0499E+00
 RMS FCTR                             2.7229E+00
 RMS FGEV                             1.2435E+01
 RMS FGR                              4.6166E+01
 RMS FGR12                            1.8572E+01
 RMS FIRA                             1.6878E+01
 RMS FIRA_R                           1.9999E+01
 RMS FIRE                             1.6878E+01
 RMS FIRE_R                           1.9999E+01
 RMS FSA                              8.2223E+00
 RMS FSH                              2.4408E+01
 RMS FSH_G                            2.6391E+01
 RMS FSH_R                            2.9027E+01
 RMS FSH_RUNOFF_ICE_TO_LIQ            7.0127E+00
 RMS FSH_TO_COUPLER                   2.6891E+01
 RMS FSH_V                            4.5004E+00
 RMS FSM                              3.4866E+01
 RMS FSR                              8.2223E+00
 RMS FSRND                            1.6709E+00
 RMS FSRNI                            1.1787E+00
 RMS FSRVD                            2.7010E+00
 RMS FSRVI                            3.3041E+00
 RMS GSSHA                            1.5678E+04
 RMS GSSHALN                          5.4977E+02
 RMS GSSUN                            6.1749E+04
 RMS GSSUNLN                          3.8598E+03
 RMS H2OSFC                           3.4983E+00
 RMS H2OSNO                           3.4834E+03
 RMS H2OSNO_TOP                       2.0577E+00
 RMS HEAT_CONTENT1                    6.0100E+08
 RMS HIA                              1.4673E+00
 RMS HIA_R                            1.7347E+00
 RMS HUMIDEX                          1.6944E+00
 RMS HUMIDEX_R                        2.0060E+00
 RMS ICE_CONTENT1                     3.5288E+03
 RMS JMX25T                           1.0772E+01
 RMS Jmx25Z                           2.5791E+01
 RMS LAKEICETHICK                     4.4332E+00
 RMS LIQUID_CONTENT1                  4.8482E+02
 RMS RH2M                             2.6788E+00
 RMS RSSHA                            2.9115E+03
 RMS RSSUN                            1.1701E+03
 RMS SABG                             4.7163E+00
 RMS SABG_PEN                         2.8251E+00
 RMS SABV                             5.6314E+00
 RMS SMP                              4.3079E+06
 RMS SNOFSRND                         1.4745E+00
 RMS SNOTXMASS                        9.4531E+05
 RMS SNOWDP                           7.4736E+00
 RMS SNOWICE                          3.3934E+03
 RMS SNOWLIQ                          1.0161E+02
 RMS SNOW_DEPTH                       7.4780E+00
 RMS SNOW_PERSISTENCE                 2.1958E+10
 RMS SOILICE                          1.6804E+01
 RMS SOILLIQ                          3.5003E+01
 RMS SOILRESIS                        4.4121E+02
 RMS SOILWATER_10CM                   8.2505E+00
 RMS SSRE_FSR                         9.0122E+00
 RMS SSRE_FSRND                       2.0546E+00
 RMS SSRE_FSRNI                       1.3729E+00
 RMS SSRE_FSRVD                       2.7555E+00
 RMS SSRE_FSRVI                       3.3657E+00
 RMS SWBGT                            1.0088E+00
 RMS SWBGT_R                          1.1942E+00
 RMS TBUILD                           1.0563E+00
 RMS TG                               3.6807E+00
 RMS TH2OSFC                          7.3008E+00
 RMS TKE1                             2.7636E+02
 RMS TLAKE                            5.2464E+00
 RMS TOTSOILICE                       2.3593E+02
 RMS TOTSOILLIQ                       4.6546E+02
 RMS TREFMXAV                         1.9114E+00
 RMS TSA                              1.3748E+00
 RMS TSKIN                            3.5211E+00
 RMS TSL                              9.2772E+00
 RMS TSOI                             8.2627E+00
 RMS TSOI_10CM                        7.6002E+00
 RMS TSOI_ICE                         6.2370E+00
 RMS TWS                              3.4340E+03
 RMS VCMX25T                          4.8056E+00
 RMS VEGWP                            2.6884E+06
 RMS VOLR                             2.5670E+07
 RMS VOLRMCH                          1.9816E+07
 RMS Vcmx25Z                          1.3121E+01
 RMS WBT_R                            1.1215E+00
 RMS ZWT_PERCH                        1.6130E+00

ESCOMP / CTSM