Uniform stage 3 procedure & address issues raised by June and Sara

[sknox01]

@June-Skeeter & @sknox01

Below are the issues raised by June:

I ran the third stage cleaning procedures for BBS which has brough up two issues we should think about. Running this from Matlab: fr_automated_cleaning([2023 2024],{'BBS'},[1 2 7 8]); fails to output ThirdStage data in for two reasons.

1) ThirdStage_REddyProc.R has a number of variables hard coded in it, including "FCH4", which isn't measured at BBS. Commenting out FCH4 is easy enough to do for a single manual BBS run, but will prove cumbersome for automation. This is a boundary case for our group, but if there is intention to have other sites running these procedures, there are plenty of other sites that don't measure CH4. In the spirit of modularity, I feel like these hard coded variables should probably be drawn from the .ini files rather than from the code.

2) Working around issue 1 by removing the FCH4 variable, the procedure runs fine for 2024, but fails for 2023. The reason is that the default behavior setup in the matlab call to runThirdStageCleaningREddyProc(yy(1),SiteId,'fast',2) defaults to using the current year and data from the previous year, which doesn't exist for the first year of the site. Setting to 1, it works fine. Alternatively, leaving it set to 2 and creating a dummy "pre-first year" folder by calling fr_automated_cleaning([2022],{'BBS'},[1 2]); is a hack that can work around the issue. As long as 2022/clean_tv exits and the traces are present (can be all NAN) it "works". The root of the issue I think is in the Biomet.Net/R/database_functions/read_database.R > the program crashes if any of the years requested are not available in the database for the specific site. Unless this behavior is desired/intentional for other applications, I can work out a fix for it.

Sara: add ET to the third stage.

[sknox01]

Also, consider storage terms for fluxes.

[znesic]

I have a vague recollection that June and I talked about #2 and that this was not an issue as long as the year when the site was established is defined properly in the BBS_ThirdStageini.R file: # Year the site was established estYear <- 2023_

[June-Skeeter]

@sknox01 @znesic > I have added functional stage 3 files for: BB, BB2, BBS. You'll notice that the three files area almost identical except that they contain different station metadata and the BBS file has some minor adjustments due to the lack of CH4 data.

• In my opinion having separate stage 3 ini files for each site is risky because it could lead to unnecessarily/unintentionally divergent procedures by site. What could be more appropriate is to have a default set of execution parameters that are only modified on an as needed basis (e.g. different set of RF gap filling inputs for estimating FCH4 a site with/without water table depth).
• Its relatively straightforward to have a global configuration that is updated with site specific settings. I've implemented a bare bones version of that already see the "configure" function in the updated ThirdStage.R file which take some defaults from a global _config.yml and then appends the site's third stage ini file. As is, most paramters are being draw from site specific ini files, because I didn't want to diverge too far from the existing procedures without further discussion first.

[June-Skeeter]

I think we can consider this closed as the gist of it has been implemented.