Why cross validate in Evaluate when there's no (re)training?

[tristndev]

As far as I know, cross validation is used in this way:

repeat k times:
     split data (e.g. randomly) into train and test data
     train model on train data
     evaluate trained model on test data
     save metric (e.g. accuracy)

However, looking at the code in the Evaluate step, I don't see any re-training happening there. The data is just split up and the model (trained in the previous Scan phase) is evaluated on those subsets of my testing data (which doesn't seem senseful to me).

I am pretty sure this is not the standard way of doing it but am happy to be wrong. Looking forward to your responses.

[mikkokotila]

Consider the following workflow. You start with one dataset consisting of 50,000 samples/rows. You split it into half, and the first half (25,000 samples) you use as your train and validation data for the experiment. Once you have completed the experiment, you feed the other half (25,000 samples) into Evaluate.

I would not use evaluate with the same data you have already used in the experiment, because now there is bias in the model towards that data.

Here we are left with the question of why not also do cross-validation as part of the training process. This is something I've looked at, and it is not straightforward to do with tensors for each epoch.

Closing here. Feel free to open new issue if anything.

[tristndev]

I am completely with you: we need to split up our data into training and testing sets. Training Data goes into Scan, Test Data goes into Evaluate.

I just don't see a reason why you would cross validate in Evaluate without any training happening. You understand what I mean?