Need to fix: The first plot the user sees in Tyra displays the average metric over test sets, rather than the metric for the test set at that given time.
For example, if you train a model through December 31, 2012, and then test that model on 2012, then 2013, then 2014, and finally 2015, the current implementation plots a point at January 1, 2015 (the start of the final test set) using the average value across those test sets.
The current implementation hides that nuance and reduces variance in performance.
Need to fix: The first plot the user sees in Tyra displays the average metric over test sets, rather than the metric for the test set at that given time.
For example, if you train a model through December 31, 2012, and then test that model on 2012, then 2013, then 2014, and finally 2015, the current implementation plots a point at January 1, 2015 (the start of the final test set) using the average value across those test sets.
The current implementation hides that nuance and reduces variance in performance.