What is the evaluation significance of model training?

[zhouhao-learning]

I saw the model in compilation, metrics = [vamp. metric_VAMP, vamp. metric_VAMP2], and then I saw the training visualization as follows:

But I don't know how to analyze this graph, what does the X and Y axes mean, and how to judge whether the curve is good or bad?

[amardt]

Hey,

this is a so called training curve of your model which tells you how your model is performing during different training stages. The X axes tells at which epoch you are in the training and the Y axes the score (Vamp1 of Vamp2). You see the curve for training and validation data. The validation score tells you how good your model is generalizing to unseen data points (which is in the end what your are aiming for). The higher the score the better is your model. However, with the vampnet method you can not directly compare models with different tau, because the vamp score depends on it. Additionally, if you have different output sizes you have to make sure to just compare the top k (k must be smaller or equal to the smallest output size.) singular values, vampnet provides an option to set top_k.

Best, Andreas

[zhouhao-learning]

@amardt I see what you're saying, you mean, this is like the val_acc in supervised learning, the higher the better, but the range is not less than or equal to 1

[amardt]

Hey, so in principle you get for each process in your data that the model is able to resolve a value between 0 and 1. If you have 2 output nodes you can just resolve one process, if you have 3 two processes and so on. So the score lies always between 0 and n-1 where n is the number of output nodes of the last layer of your model.

Best Andreas

[zhouhao-learning]

@amardt Dear If you say that the x-axis represents the epoch of the current training, then this is wrong, because I set nb_epoch = 10, but the value of the x-axis in the figure is obviously more than 10, is my understanding wrong??

In addition, you say "So the score lies always between 0 and n-1 where n is the number of output nodes of the last layer of your model." I don't quite understand what you mean, you mean, the y-axis The scope depends on how many output_size I have?? If this is the case, then I set output_size=4, then the maximum value of my y-axis should be 3, but the maximum value of the above diagram y is less than 2. Is this because the model has no good reason for training? Or am I getting it wrong?

[amardt]

Hey, I don't know how your code looks like, but if you copied the code from our examples you set the epochs and then train the model for different losses with this number of epochs. So you get in total your epochs times number of losses you use.

The y-axis is the score yes. And with output size = 4 the score will lie for all possible data between 0 and 3. However, it depends on your data how high this value can get. As I said the values represent processes. The slower the process the closer to 1 is the value. So you cannot absolutely say how high your model can get on your data, just that the higher the better.

For a deeper understanding of Markov State Models and how the singular/eigen values are connected to the slowness of processes I recommend to read the published papers about it.

Best,

Andreas