Run Hypothesis Test(s) On Verified `cfa-forecasttools` Output

[AFg6K7h4fhy2]

This issue has been discussed verbally in recent history; nonetheless, being made explicit in this fashion allows others to provide their thoughts and list considerations for this task

Once https://github.com/CDCgov/pyrenew-flu-light/issues/6 is completed, the Kolmogorov-Smirnov test, possibly in addition to other statistical tests, can be used for evaluating output files from pyrenew-flu-light and cfaepim having been run with the same data and more or less the same parameters. At present, the author intends to use some code incorporating SciPy's kstest (https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kstest.html) for this task.

This issue's author would appreciate any feedback or examples from CFA members for where what should be the same models have been jointly evaluated in Python. Once the author completes this task, the author will incorporate any comments or reflections produced into a lessons document on hang-ups in the process of porting over and evaluating cfaepim in PyRenew for other CFA members.

[AFg6K7h4fhy2]

@SamuelBrand1 Any suggestions for additional comparison procedures, beyond visualizations (which would include QQ plots & credible intervals) and the two sample KS test? Do you think I should also use the two sample Cramér–von Mises criterion (and or maybe even a modified Anderson–Darling test)? Suggestions would be appreciated.

Reminder of what the output looks like (I read this into a dataframe):

2024-03-30,wk inc flu hosp,-1,2024-03-23,02,quantile,0.2,6
2024-03-30,wk inc flu hosp,-1,2024-03-23,02,quantile,0.25,7
2024-03-30,wk inc flu hosp,-1,2024-03-23,02,quantile,0.3,7
2024-03-30,wk inc flu hosp,-1,2024-03-23,02,quantile,0.35,8
2024-03-30,wk inc flu hosp,-1,2024-03-23,02,quantile,0.4,8
2024-03-30,wk inc flu hosp,-1,2024-03-23,02,quantile,0.45,9
2024-03-30,wk inc flu hosp,-1,2024-03-23,02,quantile,0.5,10

[AFg6K7h4fhy2]

These are the implementations of these tests I am interested in using:

• ks_2samp, scipy (see here)
• cramervonmises_2samp, scipy (see here)
• anderson_ksamp, scipy (see here)

Adding DHM as well here @dylanhmorris. Post-2024-09-06 meeting this is slightly less prioritized, possibly conditional on non-completion before certain threshold date.