search

E3SM-Project / E3SM

Energy Exascale Earth System Model source code. NOTE: use "maint" branches for your work. Head of master is not validated.
https://docs.e3sm.org/E3SM
Other
345 stars 352 forks source link

ELM crashes with 1-d output (again) #2193

Open bbye opened 6 years ago

bbye commented 6 years ago

I seems that ELM has lost the ability to output in 1-d. It works fine for 2-d, but when setting hist_dov2xy = .false. in the user_nl_clm file I get the following error when the model tries to output a history file:

[0] pio_support::pio_die:: myrank= -1 : ERROR: [180] NetCDF: Numeric conversion not representable [0] pionfwrite_mod::write_nfdarray_double: 250 :

Here is the stack-trace:

[432] Image PC Routine Line Source
[432] e3sm.exe 000000000133E8FB Unknown Unknown Unknown [432] e3sm.exe 0000000001087681 pio_support_mp_pi 120 pio_support.F90 [432] e3sm.exe 00000000010857E2 pio_utils_mp_chec 74 pio_utils.F90 [432] e3sm.exe 0000000001192AC9 pionfwrite_mod_mp 250 pionfwrite_mod.F90.in [432] e3sm.exe 0000000001161660 piodarray_mp_writ 645 piodarray.F90.in [432] e3sm.exe 0000000001163BD0 piodarray_mp_writ 221 piodarray.F90.in [432] e3sm.exe 000000000115F0C9 piodarray_mp_writ 291 piodarray.F90.in [432] e3sm.exe 0000000000629D50 ncdio_pio_mpncd 1811 ncdio_pio.F90.in [432] e3sm.exe 00000000005D2087 histfilemod_mp_hf 2711 histFileMod.F90 [432] e3sm.exe 00000000005C65C1 histfilemod_mp_hi 3190 histFileMod.F90 [432] e3sm.exe 00000000005059C6 clm_driver_mp_clm 1285 clm_driver.F90 [432] e3sm.exe 00000000004F4E40 lnd_comp_mct_mp_l 464 lnd_comp_mct.F90 [432] e3sm.exe 0000000000430D84 component_modmp 728 component_mod.F90 [432] e3sm.exe 00000000004172A5 cime_comp_modmp 2650 cime_comp_mod.F90 [432] e3sm.exe 0000000000430AAC MAIN__ 103 cime_driver.F90 [432] e3sm.exe 000000000041519E Unknown Unknown Unknown [432] libc-2.12.so 00002B826AB8BD1D __libc_start_main Unknown Unknown [432] e3sm.exe 00000000004150A9 Unknown Unknown Unknown

I used the command: ./create_newcase -case hist_outputtest02 -compset I1850CRUCLM45CN -res f19_f19 -project ACME -mach anvil

I should note that I have tried several configurations and they always fail for 1-d output except for one point simulation (the 1x1_smallvilleIA.CLM45CNCROP compset)

acme-y9s commented 6 years ago

@bbye I tried to output some selected PFT(1-d) level variables, it worked for me. Please check my below setting. hist_fincl1 = 'COL_FIRE_CLOSS','COL_FIRE_NLOSS','DWT_CLOSS','DWT_NLOSS','PROD100C','PROD100N','PROD10C','PROD10N','PRODUCT_CLOSS','PRODUCT_NLOSS' hist_fincl2 = 'GPP','NEE','NEP','NPP','TLAI','TOTVEGC','TOTVEGN','FROOTC','FROOTN','LIVECROOTC','LIVECROOTN','DEADCROOTC','DEADCROOTN','LIVESTEMC','LIVESTEMN','DEADSTEMC','DEADSTEMN','TOTPFTC','TOTPFTN','PFT_FIRE_CLOSS','PFT_FIRE_NLOSS' hist_dov2xy = .true., .false. hist_mfilt = 1,1 hist_nhtfrq = 0,0 But my job failed when I added 1-d column level variables to user_nl_clm, my setting is

hist_fincl1 = 'COL_FIRE_CLOSS','COL_FIRE_NLOSS','DWT_CLOSS','DWT_NLOSS','PROD100C','PROD100N','PROD10C','PROD10N','PRODUCT_CLOSS','PRODUCT_NLOSS' hist_fincl2 = 'GPP','NEE','NEP','NPP','TLAI','TOTVEGC','TOTVEGN','FROOTC','FROOTN','LIVECROOTC','LIVECROOTN','DEADCROOTC','DEADCROOTN','LIVESTEMC','LIVESTEMN','DEADSTEMC','DEADSTEMN','TOTPFTC','TOTPFTN','PFT_FIRE_CLOSS','PFT_FIRE_NLOSS' hist_fincl3 = 'TOTLITC','TOTLITN','CWDC','CWDN','DWT_CLOSS','DWT_NLOSS','HR','TOTCOLC','TOTCOLN','TOTECOSYSC','TOTECOSYSN','TOTSOMC','TOTSOMN' hist_dov2xy = .true., .false., .false. hist_type1d_pertape = '','PFTS','COLS' hist_mfilt = 1,1,1 hist_nhtfrq = 0,0,0

From your setting, it seems to me you make all default variables to 1-d output. Considering both your case and my case, maybe there are only some column level related variables will fail to be outputted in 1-d.

bbye commented 3 years ago

It's been awhile since this issue has been updated, but I narrowed down the variable that is causing the crash - SOILLIQ (which is h2osoi_liq in ELM). I don't have any more details, nor do I know which part of the variable is causing the rouge values. I tried some print statements but all the values looked OK, at least for the active grid cells. The urban values are all 1e75, which seemed weird, but I don't know anything about the urban model so I don't know what they should be.