search

parksw3 / epidist-paper

Other
10 stars 4 forks source link

Additional proper scoring rules #50

Open seabbs opened 1 year ago

seabbs commented 1 year ago

@sbfnk suggests adding another proper scoring rule into the mix such as the absolute or relative error (to allow comparison of point estimate vs distribution performance). Another alternative would be to use the Weighted interval score approximation to the CRPRS and then show the decomposition into over/under prediction + sharpness.

On the face of it this all makes sense. We are a bit space constrained, already show relative performance, and have just quite a lot going on. That being said I think it would make sense to try adding and if we think its overwhelming put into the SI and just discuss and its useful information.,

parksw3 commented 1 year ago

I think this is a good idea, especially since other people will be more familiar with things like absolute errors.

We are a bit space constrained...

I also agree with this. And also agree with putting into the SI.

I had some summary statistics that I wanted to see as well and so will take a crack at it.

parksw3 commented 1 year ago

Here are three plots showing bias (the more classical one), RMSE, and coverage. I think these three plots would be pretty intuitive for most people? I feel like fancier summary statistics might be a bit more difficult for broader audience. So I think it's a good idea to show the traditional ones like these.

exponential_bias

exponential_rmse

exponential_coverage

seabbs commented 1 year ago

Just realised I didn't respond to this with actual words. These are nice and we should put them in. Not sure how we should compare for space etc but lets give it a whirl