search

nci / scores

Metrics for the verification, evaluation and optimisation of forecasts, predictions or models.
https://scores.readthedocs.io/
Apache License 2.0
65 stars 17 forks source link

Key Features page in docs - follow up questions #569

Closed Steph-Chong closed 1 month ago

Steph-Chong commented 4 months ago

See new "Key Features of scores" page in the docs: https://scores.readthedocs.io/en/develop/key_features.html

Tennessee has merged the new key features page to develop. He asked me to open a new issue with the remaining follow up questions.

The primary purpose of the page is for people to easily see the key features - the hope is this may help potential new users to work out if scores is likely to be useful to them.

  1. Are there any other major key features/"selling points" of scores that should be called out on the page? (Note: I am not trying to add things to the page for the sake of adding things to the page - there is already plenty on it. I am asking in case there is something worth adding that is not currently included).

  2. At present, isotonic regression (reliability diagrams) are not mentioned. Is this a sufficiently strong key feature/"selling point" that it should be mentioned?

  3. In the sub-dot-points beneath "Over 50 metrics, statistical techniques and data processing tools, including:", tools for pre-processing data are not currently mentioned. (a) Should tools for pre-processing data be mentioned? (b) If yes, perhaps something along the lines of "tools for pre-processing data - including data matching, discretisation and cumulative density function manipulation."? (I am open to any suggestions for modifications or alternative phrasing).

  4. Re. the sub-dot-point "complex scores (e.g., threshold-weighted continuous ranked probability score (twCRPS))": should the brackets include "families of consistent scoring functions for quantiles and expectiles" either instead of or in addition to twCRPS?

  5. Not a question - but any feedback on the entire page is welcome.

tennlee commented 3 months ago

ping @nicholasloveday I'm happy with this page as-is, but it would be fine to make some of the suggestions Steph has made also. Just let us know what you think.

Steph-Chong commented 3 months ago

I have since had some additional thoughts.

I think it might be worth mentioning the following two things (I haven't work-shopped the language, I just typed these out quickly):

Also, is it additionally worth mentioning that scores includes consistent and threshold weighted scores?

nicholasloveday commented 3 months ago

Regarding questions 2 and 4, I suggest changing the "complex scores..." line to "recently developed, user-focused scores and diagnostics including threshold weighted scores (e.g., threshold weighted continuous ranked probability score (twCRPS) (Gneiting and Ranjan 2011) and threshold weighted mean squared error (MSE) (Taggart 2022), Murphy diagrams (Ehm et al., 2016), and isotonic regression for reliability diagrams (Dimitriadis et al., 2021).

nicholasloveday commented 3 months ago

Regarding question 3. I don't feel like any changes need to be made

nicholasloveday commented 3 months ago

Can be used at any scale - from megabytes to petabytes

This relates to the existing line "Uses Dask for scaling and performance.", which I think is fine

Steph-Chong commented 3 months ago

Thanks for that helpful feedback @nicholasloveday. I'll update the page to incorporate it.