Observed Input and Report

At the moment we have ExcelInput to read in observed files, Report to gather predicted values from simulations and PredictedObserved to match up those data tables at the end of the simulation. However, this often leads to reports that gather a lot more variables than what is required to match the observed data, and mistakes when matching up the observed and predicted columns. Related Issues: #8413 #6807

To address this, we are working on a new PredictedObserved system that can read in an observed file, generate the required reports for matching data and output the predicted observed tables with less manual setup than currently required. We'll be working on this in parts.

Part 1

[x] An ObservedInput model that replicates the functionality of ExcelInput, but also reads all the column headings out of the file and makes them available
[x] An ObservedReport that acts like a report, but builds the list of reporting variables from the columns ObeservedInput read in
[x] ObservedReport needs to be able to take a frequency to report on so that it can replace existing reports
[x] The column names are filtered based on the observed values that the simulation has
[x] ObservedReport has a matching column name that can be used to filter the rows that are used for filtering column names
[ ] Unit tests on completed parts
[x] A solution so that we don't need to base ObservedReport on Report

Bugs

[x] Crash if Simulation has ObservedReport but no ObservedInput
[x] Crash if Simulation name doesn't match name in observed data. Simulation ID error.

Part 2

[ ] ObservedReport should replace the functionality of PredictedObserved and return a table with predicted and observed columns
- [ ] The columns should only include valid variables
[ ] Tests must be adjusted to now work on ObservedInput/ObservedReport instead of PredictedObserved
[ ] A manager script can ask the ObservedInput what is the final date for an observation given a simulation name. This would allow harvest scripts to avoid harvesting before the final observation.

Part 3

[ ] Create a flag or marker for running on Jenkins so that all data not required for doing predicted observed stats is not gathered. Such as daily values for graphing that don't have an observed value.
[ ] Don't run any other reports while this flag is active so that less data must be recorded
[ ] Looking to gain about 20% back on Jenkins validation speed.

APSIMInitiative / ApsimX

Observed Input and Report #8462