Closed anewman89 closed 4 months ago
@anewman89 Obtain a test FLUXNET2015 data file that is acceptable for testing.
When comparing the new script for computing TCI from CESM with the output from the old script, I discovered a bug here: https://github.com/dtcenter/METplus/blob/1f081dd1daed864840e695b892bc640dac573bcf/parm/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI/cesm_tci.py#L37-L46
<xarray.DataArray (season: 4, lat: 192, lon: 288)>
array([[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]],
[[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
...,
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan],
[nan, nan, nan, ..., nan, nan, nan]]])
Coordinates:
* lat (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 87.17 88.12 89.06 90.0
* lon (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
* season (season) object 'DJF' 'JJA' 'MAM' 'SON'
"JJA" actually should be 1, and "MAM" is 2. "SON" and "DJF" are correct (3 and 0 respectively). This will be fixed in the next version by using explicit season name indexing instead of relying on integers relative to the ordering.
Thanks Dan!
@DanielAdriaansen Some example raw FLUXNET data in CSV format can be found on our filesystem at /glade/work/anewman/fluxnet
@anewman89 for the raw FLUXNET data you shared at /glade/work/anewman/fluxnet
, what is a sample soil and flux variable (column) name to use for testing?
TIMESTAMP_START,TIMESTAMP_END,TA_F,TA_F_QC,SW_IN_POT,SW_IN_F,SW_IN_F_QC,LW_IN_F,LW_IN_F_QC,VPD_F,
VPD_F_QC,PA_F,PA_F_QC,P_F,P_F_QC,WS_F,WS_F_QC,WD,USTAR,RH,NETRAD,PPFD_IN,SW_OUT,LW_OUT,
CO2_F_MDS,CO2_F_MDS_QC,TS_F_MDS_1,TS_F_MDS_2,TS_F_MDS_3,TS_F_MDS_4,TS_F_MDS_1_QC,
TS_F_MDS_2_QC,TS_F_MDS_3_QC,TS_F_MDS_4_QC,SWC_F_MDS_1,SWC_F_MDS_1_QC,G_F_MDS,G_F_MDS_QC,
LE_F_MDS,LE_F_MDS_QC,LE_CORR,LE_CORR_25,LE_CORR_75,LE_RANDUNC,H_F_MDS,H_F_MDS_QC,H_CORR,
H_CORR_25,H_CORR_75,H_RANDUNC,NIGHT,NEE_VUT_REF,NEE_VUT_REF_QC,NEE_VUT_REF_RANDUNC,
NEE_VUT_25,NEE_VUT_50,NEE_VUT_75,NEE_VUT_25_QC,NEE_VUT_50_QC,NEE_VUT_75_QC,RECO_NT_VUT_REF,
RECO_NT_VUT_25,RECO_NT_VUT_50,RECO_NT_VUT_75,GPP_NT_VUT_REF,GPP_NT_VUT_25,GPP_NT_VUT_50,
GPP_NT_VUT_75,RECO_DT_VUT_REF,RECO_DT_VUT_25,RECO_DT_VUT_50,RECO_DT_VUT_75,GPP_DT_VUT_REF,
GPP_DT_VUT_25,GPP_DT_VUT_50,GPP_DT_VUT_75,RECO_SR,RECO_SR_N
This use case will now depend on and utilize the METcalcpy diagnostics submodule to compute TCI. See dtcenter/METcalcpy#364.
Outstanding questions:
1) What is the reference for the TCI equation/calculation we'd like to use in documentation?
2) Given the site-by-site nature of the observation data files, how many sites do we wish to include in the new sample data for this use case?
3) I noticed that if a user downloads data from "Ameriflux" and not "Fluxnet", despite the variables and values inside the files being the same, the file names are not the same. Do we want to support both sources? Is there a third or other sources we should support?
4) Finalize quality control filtering with @anewman89
5) The sample data for this use case needs to be changed. It's not a new use case, but the obs data needs to be swapped out and then the reference data for automated testing needs to be updated as well.
Outstanding questions:
- What is the reference for the TCI equation/calculation we'd like to use in documentation?
Use this reference for TCI calculation: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2011GL048268
- Given the site-by-site nature of the observation data files, how many sites do we wish to include in the new sample data for this use case?
Let's shoot for 20-30 global sites (FLUXNET)
- I noticed that if a user downloads data from "Ameriflux" and not "Fluxnet", despite the variables and values inside the files being the same, the file names are not the same. Do we want to support both sources? Is there a third or other sources we should support?
Easy configuration change, so a user can support any filenames as long as the variables are in the files.
- Finalize quality control filtering with @anewman89
Ended up setting QC to 80% of the finer resolution data passing in order to use the daily data, at least 1 day in an individual season must be present, and at least 10 days in all seasons at a given site must be present. These are configurable by the user.
- The sample data for this use case needs to be changed. It's not a new use case, but the obs data needs to be swapped out and then the reference data for automated testing needs to be updated as well.
Still need to iron out the sample data, hoping @georgemccabe can do this.
Closed via #2550.
Replace italics below with details for this issue.
Describe the Enhancement
The TCI use case in the current METplus User's guide section 7.2.5 (Land Surface) needs to be enhanced for several aspects.
First, the current version reads precomputed values of TCI from observations in the Python Embedding script
parm/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI/fluxnet2015_tci.py
This python embedding script will need to be upgraded to read the raw surface flux and soil moisture data from FLUXNET2015 data and compute TCI for user specified season (e.g., summer - JJA or winter - DJF).
Second, the observation data script should have user defined flexibility to compute TCI from both latent and sensible heat flux, as well as multiple soil depths.
Finally, the model data python embedding script should have more flexibility to use different soil layers to match the flexibility of the observation data python script.
parm/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI/cesm_tci.py
Time Estimate
2-3 days
Sub-Issues
Consider breaking the enhancement down into sub-issues.
Relevant Deadlines
12/31/2023
Funding Source
7790111 (CLASP)
Define the Metadata
Assignee
Labels
Projects and Milestone
Define Related Issue(s)
Consider the impact to the other METplus components.
Enhancement Checklist
See the METplus Workflow for details.
feature_<Issue Number>_<Description>
feature <Issue Number> <Description>