search

jkrasting / gfdlvitals

Scripts to generate global means (vitals) for GFDL models
Other
4 stars 6 forks source link

gfdlvitals errors out: sqlite3.OperationalError: no such column: nan #13

Closed nikizadehgfdl closed 2 years ago

nikizadehgfdl commented 2 years ago

This happens whether I do it interactively or via frepp.

 an009: /home/Niki.Zadeh % /home/mdteam/anaconda/envs/gfdlvitals_v3.0.1/bin/gfdlvitals -s 0250 -e 0250 -o /home/Niki.Zadeh/fre/ESM4.2_2021.03_chem_updates_MOM6up/ESM4p2_piControl_spinup_lowss_WOAinitCond_upSoilC_VertDiff1/gfdl.ncrc4-intel18-prod-openmp/db -g /archive/gold/datasets/OM4_05/mosaic_c96.v20180227.tar /archive/Niki.Zadeh/fre/ESM4.2_2021.03_chem_updates_MOM6up/ESM4p2_piControl_spinup_lowss_WOAinitCond_upSoilC_VertDiff1/gfdl.ncrc4-intel18-prod-openmp/history
Matplotlib created a temporary config/cache directory at /vftmp/Niki.Zadeh/pid10298/matplotlib-n3qmhm6u because the default path (/home/Niki.Zadeh/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Dmgetting files ...
Complete!
Processing 02500101
02500101.atmos_month.nc
02500101.atmos_co2_month.nc
02500101.atmos_month_aer.nc
02500101.aerosol_month_cmip.nc
02500101.land_month.nc
Traceback (most recent call last):
  File "/home/mdteam/anaconda/envs/gfdlvitals_v3.0.1/bin/gfdlvitals", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/mdteam/gfdlvitals/scripts/gfdlvitals", line 186, in <module>
    process_year(cliargs, _infile)
  File "/home/mdteam/gfdlvitals/scripts/gfdlvitals", line 114, in process_year
    gfdlvitals.models.CM4.routines(args, infile)
  File "/home/mdteam/gfdlvitals/gfdlvitals/models/CM4.py", line 51, in routines
    averagers.land_lm4.xr_average(fyear, tar, modules)
  File "/home/mdteam/gfdlvitals/gfdlvitals/averagers/land_lm4.py", line 135, in xr_average
    xrtools.xr_to_db(
  File "/home/mdteam/gfdlvitals/gfdlvitals/util/xrtools.py", line 55, in xr_to_db
    write_sqlite_data(sqlfile, var, str(fyear), str(dset[var].data))
  File "/home/mdteam/gfdlvitals/gfdlvitals/util/gmeantools.py", line 317, in write_sqlite_data
    cur.execute(sql)
sqlite3.OperationalError: no such column: nan

Or the frepp logs: /home/Niki.Zadeh/fre/ESM4.2_2021.03_chem_updates_MOM6up/ESM4p2_piControl_spinup_lowss_WOAinitCond_upSoilC_VertDiff1/gfdl.ncrc4-intel18-prod-openmp/stdout/postProcess/ESM4p2_piControl_spinup_lowss_WOAinitCond_upSoilC_VertDiff1_refineDiag_02500101.o17368398

nikizadehgfdl commented 2 years ago

Thanks to Lori's tip, I added -c ocean,ice,atmos to the command and the db files were generated. It turns out the -c land is causing this error.

jkrasting commented 2 years ago

Thanks @nikizadehgfdl and @LoriTSentman. In this experiment, the tot_soil_C field from the model output contains all NaN values. Can you confirm?

I'd like some feedback on how gfdlvitals should handle this situlation. Should it print a warning that the scalar for a variable is NaN, and move on?

nikizadehgfdl commented 2 years ago

Yes, I like the idea of a warning and let the .db generation for other components to proceed. I don't know what is causing the NaN's in model output. My run was a repeat of Lori's original xml/run with a modified COBALT code.

LoriTSentman commented 2 years ago

Thanks for following up on this. I'm not sure why this field is NaN. I'll have to defer to Sergey and Elena for assessment.

Lori

Dr. Lori T. Sentman, Research Physical Scientist Biogeochemistry, Atmospheric Chemistry, and Ecosystems Division NOAA/Geophysical Fluid Dynamics Laboratory 201 Forrestal Road, Princeton, NJ 08540 (609) 452-6563

On Fri, Feb 11, 2022 at 2:01 PM Niki Zadeh @.***> wrote:

Yes, I like the idea of a warning and let the .db generation for other components to proceed. I don't know what is causing the NaN's in model output. My run was a repeat of Lori's original xml/run with a modified COBALT code.

— Reply to this email directly, view it on GitHub https://github.com/jkrasting/gfdlvitals/issues/13#issuecomment-1036523010, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQDZ4ROYXMQEIQGTDPV7XG3U2VMHNANCNFSM5OBVQG3Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

LoriTSentman commented 2 years ago

Sergey confirmed that this output field is for the CORPSE soil carbon model. Here, we switched to the CENTURY-like soil carbon so this field is not populated. Other equivalent soil carbon fields do exist with this soil carbon model and can be used for analysis. He said it is not a problem to calculate an equivalent of tot_soil_C in the CENTURY-like carbon model and report it to the diagnostics and he can do that in a future update.

For the time being, I agree that it should return a warning but continue the .db generation for other components.

Lori

Dr. Lori T. Sentman, Research Physical Scientist Biogeochemistry, Atmospheric Chemistry, and Ecosystems Division NOAA/Geophysical Fluid Dynamics Laboratory 201 Forrestal Road, Princeton, NJ 08540 (609) 452-6563

---------- Forwarded message --------- From: Niki Zadeh @.> Date: Fri, Feb 11, 2022 at 2:01 PM Subject: Re: [jkrasting/gfdlvitals] gfdlvitals errors out: sqlite3.OperationalError: no such column: nan (Issue #13) To: jkrasting/gfdlvitals @.> Cc: Lori Thompson Sentman @.>, Mention < @.>

Yes, I like the idea of a warning and let the .db generation for other components to proceed. I don't know what is causing the NaN's in model output. My run was a repeat of Lori's original xml/run with a modified COBALT code.

— Reply to this email directly, view it on GitHub https://github.com/jkrasting/gfdlvitals/issues/13#issuecomment-1036523010, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQDZ4ROYXMQEIQGTDPV7XG3U2VMHNANCNFSM5OBVQG3Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

jkrasting commented 2 years ago

This issue has resurfaced in a slightly different way -- the plots are generated but the global mean values sometimes display "nan" in the plot legends.