Enhancement: Enhance Terrestrial Coupling Index (TCI) Use Case

anewman89 commented 10 months ago

Replace italics below with details for this issue.

Describe the Enhancement

The TCI use case in the current METplus User's guide section 7.2.5 (Land Surface) needs to be enhanced for several aspects.

First, the current version reads precomputed values of TCI from observations in the Python Embedding script parm/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI/fluxnet2015_tci.py

This python embedding script will need to be upgraded to read the raw surface flux and soil moisture data from FLUXNET2015 data and compute TCI for user specified season (e.g., summer - JJA or winter - DJF).

Second, the observation data script should have user defined flexibility to compute TCI from both latent and sensible heat flux, as well as multiple soil depths.

Finally, the model data python embedding script should have more flexibility to use different soil layers to match the flexibility of the observation data python script. parm/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI/cesm_tci.py

Time Estimate

2-3 days

Sub-Issues

Consider breaking the enhancement down into sub-issues.

[ ] FLUXNET raw data ingest for the observation data python embedding script
[ ] Modifications to observation data python embedding script and METplus configuration for user selected variables
[ ] Modifications to model data python embedding script for user selected variables
[ ] Add a description of this data into the METplus Verification Datasets Guide.

Relevant Deadlines

12/31/2023

Funding Source

7790111 (CLASP)

Define the Metadata

Assignee

[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required

Labels

[ ] Select component(s)
[ ] Select priority
[ ] Select requestor(s)

Projects and Milestone

[ ] Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
[ ] Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

[ ] METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding Source.
[ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
[ ] Complete the development and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Add any new Python packages to the METplus Components Python Requirements table.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
[ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Development issues Select: Repository level development cycle Project for the next official release Select: Milestone as the next official version
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Close this issue.

anewman89 commented 10 months ago

@anewman89 Obtain a test FLUXNET2015 data file that is acceptable for testing.

DanielAdriaansen commented 10 months ago

When comparing the new script for computing TCI from CESM with the output from the old script, I discovered a bug here: https://github.com/dtcenter/METplus/blob/1f081dd1daed864840e695b892bc640dac573bcf/parm/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI/cesm_tci.py#L37-L46

<xarray.DataArray (season: 4, lat: 192, lon: 288)>
array([[[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]]])
Coordinates:
  * lat      (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 87.17 88.12 89.06 90.0
  * lon      (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
  * season   (season) object 'DJF' 'JJA' 'MAM' 'SON'

"JJA" actually should be 1, and "MAM" is 2. "SON" and "DJF" are correct (3 and 0 respectively). This will be fixed in the next version by using explicit season name indexing instead of relying on integers relative to the ordering.

anewman89 commented 10 months ago

Thanks Dan!

anewman89 commented 7 months ago

@DanielAdriaansen Some example raw FLUXNET data in CSV format can be found on our filesystem at /glade/work/anewman/fluxnet

DanielAdriaansen commented 6 months ago

@anewman89 for the raw FLUXNET data you shared at /glade/work/anewman/fluxnet, what is a sample soil and flux variable (column) name to use for testing?

TIMESTAMP_START,TIMESTAMP_END,TA_F,TA_F_QC,SW_IN_POT,SW_IN_F,SW_IN_F_QC,LW_IN_F,LW_IN_F_QC,VPD_F,
VPD_F_QC,PA_F,PA_F_QC,P_F,P_F_QC,WS_F,WS_F_QC,WD,USTAR,RH,NETRAD,PPFD_IN,SW_OUT,LW_OUT,
CO2_F_MDS,CO2_F_MDS_QC,TS_F_MDS_1,TS_F_MDS_2,TS_F_MDS_3,TS_F_MDS_4,TS_F_MDS_1_QC,
TS_F_MDS_2_QC,TS_F_MDS_3_QC,TS_F_MDS_4_QC,SWC_F_MDS_1,SWC_F_MDS_1_QC,G_F_MDS,G_F_MDS_QC,
LE_F_MDS,LE_F_MDS_QC,LE_CORR,LE_CORR_25,LE_CORR_75,LE_RANDUNC,H_F_MDS,H_F_MDS_QC,H_CORR,
H_CORR_25,H_CORR_75,H_RANDUNC,NIGHT,NEE_VUT_REF,NEE_VUT_REF_QC,NEE_VUT_REF_RANDUNC,
NEE_VUT_25,NEE_VUT_50,NEE_VUT_75,NEE_VUT_25_QC,NEE_VUT_50_QC,NEE_VUT_75_QC,RECO_NT_VUT_REF,
RECO_NT_VUT_25,RECO_NT_VUT_50,RECO_NT_VUT_75,GPP_NT_VUT_REF,GPP_NT_VUT_25,GPP_NT_VUT_50,
GPP_NT_VUT_75,RECO_DT_VUT_REF,RECO_DT_VUT_25,RECO_DT_VUT_50,RECO_DT_VUT_75,GPP_DT_VUT_REF,
GPP_DT_VUT_25,GPP_DT_VUT_50,GPP_DT_VUT_75,RECO_SR,RECO_SR_N

DanielAdriaansen commented 5 months ago

This use case will now depend on and utilize the METcalcpy diagnostics submodule to compute TCI. See dtcenter/METcalcpy#364.

DanielAdriaansen commented 5 months ago

Outstanding questions: 1) What is the reference for the TCI equation/calculation we'd like to use in documentation? 2) Given the site-by-site nature of the observation data files, how many sites do we wish to include in the new sample data for this use case? Screenshot 2024-04-05 at 2 29 22 PM 3) I noticed that if a user downloads data from "Ameriflux" and not "Fluxnet", despite the variables and values inside the files being the same, the file names are not the same. Do we want to support both sources? Is there a third or other sources we should support? 4) Finalize quality control filtering with @anewman89 5) The sample data for this use case needs to be changed. It's not a new use case, but the obs data needs to be swapped out and then the reference data for automated testing needs to be updated as well.

DanielAdriaansen commented 5 months ago

Outstanding questions:

What is the reference for the TCI equation/calculation we'd like to use in documentation?

Use this reference for TCI calculation: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2011GL048268

Given the site-by-site nature of the observation data files, how many sites do we wish to include in the new sample data for this use case?

Let's shoot for 20-30 global sites (FLUXNET)

I noticed that if a user downloads data from "Ameriflux" and not "Fluxnet", despite the variables and values inside the files being the same, the file names are not the same. Do we want to support both sources? Is there a third or other sources we should support?

Easy configuration change, so a user can support any filenames as long as the variables are in the files.

Finalize quality control filtering with @anewman89

Ended up setting QC to 80% of the finer resolution data passing in order to use the daily data, at least 1 day in an individual season must be present, and at least 10 days in all seasons at a given site must be present. These are configurable by the user.

The sample data for this use case needs to be changed. It's not a new use case, but the obs data needs to be swapped out and then the reference data for automated testing needs to be updated as well.

Still need to iron out the sample data, hoping @georgemccabe can do this.

DanielAdriaansen commented 4 months ago

Closed via #2550.

dtcenter / METplus