search

jarad / FluSight

An R package containing functions used in the CDC Flu Forecasting competition
GNU General Public License v3.0
12 stars 7 forks source link

Alternative method to generate point forecasts #16

Closed craigjmcgowan closed 7 years ago

craigjmcgowan commented 7 years ago

Currently the point forecasts are generated from the expected value based on the probabilities assigned. Do we want to give teams another option to simply pick the individual bin with the highest probability? If they're getting very skewed probability curves from their models this method could be preferred and it shouldn't be too hard to add to the function.

nickreich commented 7 years ago

I thought point forecasts were being delivered/specified by the teams? I definitely think having something other than the expected value would be good. Theoretically, the median value is the choice that would minimize the mean absolute error, but that would be different from picking the mode, as suggested. Finding the estimated median would require finding the bin that, when the bins are ordered by week, tips the total cumulative probability over 0.5.

craigjmcgowan commented 7 years ago

Yes, they are provided by the team - this function is just provided as a way for a team to generate the point forecasts if that team's model is only producing the probabilistic outcomes. Perhaps all three options (mean, median, and mode) could be provided and the user could simply pick which they prefer to use.

nickreich commented 7 years ago

Seems fine, although I vote for default being median since we are using mean absolute error as a metric.

Nicholas G. Reich, Ph.D. Assistant Professor Department of Biostatistics and Epidemiology School of Public Health and Health Sciences University of Massachusetts, Amherst

413.545.4534 nick@schoolph.umass.edu http://reichlab.github.io/ on twitter: @reichlab On 2 Nov 2016, at 11:14, Craig McGowan wrote:

Yes, they are provided by the team - this function is just provided as a way for a team to generate the point forecasts if that team's model is only producing the probabilistic outcomes. Perhaps all three options (mean, median, and mode) could be provided and the user could simply pick which they prefer to use.

You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/jarad/FluSight/issues/16#issuecomment-257895358

jarad commented 7 years ago

Good call Nick. I wasn't thinking about the scoring metric when I coded this up.