`evaluate!` mutating?

[roland-KA]

I have a general question concerning evaluate!: As the exclamation mark in it's name shows, this is a mutating function. But it's purpose is the calculation of different measures, which basically (at least from an abstract point of view) shouldn't require to mutate the data given to evaluate!?

So, why is it nonetheless defined as being mutating? Are there calculations of specific measures which require to mutate the data (so my assumption is wrong) or is it just a way to give the implementors of this function more degrees of freedom on how to implement it?

[ablaom]

Data is not mutated. Other stuff gets mutated, eg learned parameters, which are stored in the machine after training. When you evaluate! you run a loop in which the model gets trained on different views of the data (corresponding to different folds in CV(), for example). So internally you have something like fit!(mach, rows=...) where rows is changing, and these calls mutate the learned parameters (and other stuff that implements warm/cold restart behaviour for iterative models, for example).

Is that clearer now?

https://alan-turing-institute.github.io/MLJ.jl/dev/machines/#Machines

[roland-KA]

Ok, that makes it much clearer to what happens behind the scenes of evaluate!. 👍

... but leads to a follow-up question 😊:

It was my initial understanding that evaluate! does an implicit fit! (as you explained) ... until I came across this code in part 02 (section 'Step 4. Evaluate model performance') of the tutorial:

let
  train, test = partition(eachindex(yIris), 0.7)
  mach = machine(model, XIris, yIris)
  evaluate!(mach, resampling=CV(nfolds=6),
            measures=[cross_entropy, brier_score],
            rows=train)     # cv estimate, resampling from `train`
  fit!(mach, rows=train)    # re-train using all of `train` observations
  predict(mach, rows=test); # and predict missing targets
end

The fit! immediately after evaluate! made me doubt my understanding. I thought that fit! is not necessary here, because evaluate! should have done it already.

But with your explanation and a second glimpse on it, I understand now, that the fit! is done to include all data in train whereas evaluate! uses in each of its training steps only 5/6 of the train data (because of the CV applied). Is that correct?

[ablaom]

Yes, that is correct.