Add distance measurement between forecasts

[widforss]

This is a way to compare avalanche bulletins. It is a kind of distance measurement. It is commutative but does not fulfill the triangle inequality, as the null forecast is defined as a plane.

It works in three stages. First it measures the difference between avalanche problems. Then it measures the difference between ordered lists of avalanche problems. This is called the p_score. The last stage is to compare the tuple (danger_level, emergency_warning, p_score).

It is currently rather unbalanced. This will be adjusted later.

[Jakidxav]

Hi, Aron. This looks really interesting. Is the goal to use this as a custom machine learning scoring metric later down the line, like for deep learning models perhaps? And if so, wil you be trying to make the new distance metric parabolic so that it can be optimized over?

[widforss]

Yes, that's one of the thoughts. My primary objective right now is to be able to compare the efficacy between different models with regard to the whole forecast, instead of single attributes. (The reason we cannot use a simple norm of a vector is that I want the distance between two forecasts with the right avalanche problems but with the problems out of order to be rather small.)

This is a first implementation, and currently emergency_warning's impact is too great. The next step is to weight each dimension by it's inverse variance, so the dimensions have about the same impact.

I'm currently writing to my uni's math department to see if they know of some pitfalls to avoid.

Could you elaborate on how to make the distance measurement parabolic, I'm not sure I'm following. What would be required? Do you know if you would need the triangle inequality to be true for the measurement to be able to use it?

[Jakidxav]

I am not as familiar with the triangle inequality, other than the basic x + y <= z. As it is a distance metric, my only concern is that the metric should not be square-rooted. That is, for a deep learning model the cost function J should (ideally) have a global minimum to optimize for during backpropagation: https://www.deeplearning.ai/ai-notes/optimization/

I did however find this post about a common loss metric, cross-entropy, which mentions that it does not fulfill the triangle inequality: https://medium.com/analytics-vidhya/a-friendly-introduction-to-cross-entropy-for-machine-learning-b4e9f2b1f6 The article was a little above me, but hopefully it is useful to you!

[widforss]

Ok, I think I understand what you mean now. As it shows the "distance" to the correct forecast, it will have a single global minimum, and is symmetric around this point.

[widforss]

@Jakidxav I've adjusted the metric to have a positive second derivative now. Score.calc() returns a DataFrame with the metric at the column ("", "score"). I'll merge this with master tonight.