prio-data / prediction_competition_2023

Code for generating benchmark models and evaluation scripts for the 2023 VIEWS prediction competition
4 stars 6 forks source link

Necessary improvements to the maps and line plots as presented in Berlin #26

Open hhegre opened 11 months ago

hhegre commented 11 months ago

Hi y'all,

I have reviewed the maps and figures presented in Berlin. They all look wonderful, but the following changes have to be made to make them truly excellent:

Maps for actual fatalities, cm: Range of color should go up to 100,000; we have observations of that magnitude. Great that the colours are similar across maps! Not sure how the color scheme was set up - I think there has to be more color variation in the range 0-100, it seems oddly compressed in that range. 1,000 monthly deaths should have orange color. What is the definition here? Elsewhere the gradation seems to be mapped to the log of the metric, that seems more logical to me. Font size for labels must be much larger.

Maps for actual fatalities, cm: Range of color should perhaps go up to 10,000 for pgm, but perhaps it works better if it ends ast 1,000 as you suggest. Here the color scheme is a clear log linear transformation, why is this different from cm? Labels must be much larger.

Maps for errors in predictions, CRPS and MIS, cm: Font size for labels must be much larger. There has to be labels across the entire line. I suggest (0, 1, 3, 10, 30, 100, 300, 1000). What proportion of the observations have errors larger than 1,000 - should we go to 10,000? Based on the share of countries with yellow it seems that that would be better (particularly for MIS, e.g. the CSO submission….). What is the definition of the color scheme? The gradation mapped to log of error? r.

Maps for errors in predictions, IGN, cm: Font size for labels must be much larger. Selection of labels seems to be good here. 10 is some sort of maximum value, isn’t it?

Maps for errors in predictions, CRPS and MIS, pgm: Font size for labels must be much larger. Drop decimals for numbers from 1 and upwards Here, the color mapping seems ok, although it might perhaps be even better if 1,000 was the explicit maximum so that there is a label for that (or that the maximum is labeled, too, “500” or whatever the end point is)

Maps for errors in predictions, IGN, cm: Font size for labels must be much larger. Selection of labels should be same as for cm, and 0 and 10 has to included as labels.

Map of best model by metric: Font size of labels must be much larger. I made a hack in the overleaf version…

Line graphs with percentiles Some of the graphs start at month 455 (e.g. Brandt), they should start at 457. What happens here? Font size for all labels must be much larger (except for the one identifying the figure at the top, that one can be small) No need to have “outcome” with the percentiles in the legend, the percentile is sufficient. So labels should be “1%”, “5%”, …”99%”, “mean”, and then “actuals”. By shortening font size can be larger, and short and readable is better . “Outcome” is ambiguous in any case…

kvelleby commented 11 months ago

On the color scheme, perhaps the scheme viridis "turbo" would work well for our purposes (with the blue being to zero). Gist to matplotlib-friendly version

kvelleby commented 11 months ago

On why some line-graphs starts at month 455, that might be because some submissions had errors and were updated relatively late. The plots were just not updated after getting new data.

sarakallis commented 11 months ago

Today's progress (all plots in dropbox)

It's around 1-3% of observations. I changed it to 10,000, but can easily change it back.

What is the definition of the color scheme? The gradation mapped to log of error? r.

Not sure what is meant by this question.

Yes

For log-transformed MIS and CRPS, 100 is the max in most cases, so I manually added the tick labels (0, 1, 3, 10, 30, 100). There is currently a large gap between 0 and 1, please advise whether to add more ticks in-between (see example below). I can also adjust the linthresh (currently = 0.1) to 1, which moves the 1 tick more to the left (and plots values from 0-1 linearly, and above 1 in log)

Screenshot 2023-10-16 at 17 37 11