search

rfechner / generic-hmm

Apache License 2.0
2 stars 1 forks source link

How do we measure success for the resulting model? #6

Closed rfechner closed 2 years ago

rfechner commented 2 years ago

Which metric should be used in order to make the model comparable? For which category of prediction do we use which metric?

rfechner commented 2 years ago

The Logarithmic Loss Function should probably be the way to go, since we are dealing with predictions of distributions over states.

rfechner commented 2 years ago

k-fold cross validation should be another interesting way of leasuring validity of the model. It is used in https://www.sciencedirect.com/science/article/abs/pii/S0020025515007057 to compute the F1 Score of a n-order HMM

rfechner commented 2 years ago

Interesting is the measurement of distance between sequences, e.g. the Levenshtein distance. Thus, we would not compute Precision and Recall for the F1 Score, but be able to give a rough estimate of the general performance of the model.

Since we are operating on a graph, maybe we should give the average distance of prediction instead of a binary true - false measurement of prediction