The current interworking between candiates (things one want to optimize, typically the model) and fitness strategies (what one wants to optimize) is less than beautiful. Much of it is thanks to the instrumentation-API which is the pattern for e.g. TimeFitness where one as a byproduct of the training also wants to get fitness in terms of how long it took to train.
I think that removing the assumption that one always wants to train the model a little bit before evaluating fitness (e.g. to implement this) can help clear this out into a nicer and more easy to work with design.
In particular having the fitness strategy separate from the candidate seems like an attractive way forward. I'm thinking that an API like fitness(fs::FitnessStrategy, c::AbstractCandidate) or perhaps even fitness(fs::FitnessStrategy, c::AbstractCandidate, generationnr::Int) i.e tightening up the fitness API so it no longer just takes a function. The function was only used for one single fitness strategy anyways as the instrumentation made it impossible to make any assumptions about it. The fitness strategy would then just fetch what it needs from the candidate (e.g. the model) to compute the fitness. Hopefully this removes alot of the implicit metrics where a fitness strategy just to stores the byproduct of some other operation as a mutable state which it then returns when which in turn means that fitness is easier to use and does not need to be associated with a certain candidate.
To optimize some other aspect than the model itself, one would then implement new candidate types which have the stuff one wants to optimize as members (currently we have models and optimizers). Fitness strategies would then need to throw an error if the candidate does not provide the data needed to compute the fitness. I think it is safe to say that for something to be meaningful to search for, it has to affect the fitness somehow, although the implementation might not have to be direct. I think that even creating new candidates with mapcandidate can be done somewhat automatically in the same way Functors.jl work, but perhaps I'll go for something simpler with an abstract type for "root" candidates which will have all their fields mapped (non-root candidates just pass the mapping on to their wrapped candidiate).
This means working out other solutions for
Training: This now becomes an own fitness strategy. I'm thinking it should have NaNGuard built in and probably wrap another fitness strategy which calculates the fitness after training. This is slightly awakward as the training step is not the exact fitness one is after, but it makes sense to say "my fitness metric is the accuracy on validationset after training for x iterations" so I think it is ok.
fitness caching: I think current CacheCandidate is sufficient. Perhaps use an even more explicit method, like passing a fitness vector to evolve is even more elegant, although I do like the option to look at last evaluated fitness of serialized candidates.
TimeFitness: I think this should just wrap another fitness strategy which it calculates the time for
SizeFitness: No problem?
TrainingAccuracy: Not sure, perhaps ditch it or incorporate it in the training step but default is to ignore it. Maybe training always produces the training accuracy and does not wrap another fitness strategy?
I think it is preferable if wrapping fitness strategies (e.g. TimeFitness) return all fitnesses as a (named?)tuple. This would pave the way for multi-objective optimization. Question is just how much headache it is to combine them? Does one want the tuples nested or not? Perhaps this requires some fitness combiner which just takes an arbitrary method which the user defines based on what the wrapped fitnesses are.
The current interworking between candiates (things one want to optimize, typically the model) and fitness strategies (what one wants to optimize) is less than beautiful. Much of it is thanks to the instrumentation-API which is the pattern for e.g. TimeFitness where one as a byproduct of the training also wants to get fitness in terms of how long it took to train.
I think that removing the assumption that one always wants to train the model a little bit before evaluating fitness (e.g. to implement this) can help clear this out into a nicer and more easy to work with design.
In particular having the fitness strategy separate from the candidate seems like an attractive way forward. I'm thinking that an API like
fitness(fs::FitnessStrategy, c::AbstractCandidate)or perhaps evenfitness(fs::FitnessStrategy, c::AbstractCandidate, generationnr::Int)i.e tightening up the fitness API so it no longer just takes a function. The function was only used for one single fitness strategy anyways as the instrumentation made it impossible to make any assumptions about it. The fitness strategy would then just fetch what it needs from the candidate (e.g. the model) to compute the fitness. Hopefully this removes alot of the implicit metrics where a fitness strategy just to stores the byproduct of some other operation as a mutable state which it then returns when which in turn means that fitness is easier to use and does not need to be associated with a certain candidate.To optimize some other aspect than the model itself, one would then implement new candidate types which have the stuff one wants to optimize as members (currently we have models and optimizers). Fitness strategies would then need to throw an error if the candidate does not provide the data needed to compute the fitness. I think it is safe to say that for something to be meaningful to search for, it has to affect the fitness somehow, although the implementation might not have to be direct. I think that even creating new candidates with mapcandidate can be done somewhat automatically in the same way Functors.jl work, but perhaps I'll go for something simpler with an abstract type for "root" candidates which will have all their fields mapped (non-root candidates just pass the mapping on to their wrapped candidiate).
This means working out other solutions for
I think it is preferable if wrapping fitness strategies (e.g. TimeFitness) return all fitnesses as a (named?)tuple. This would pave the way for multi-objective optimization. Question is just how much headache it is to combine them? Does one want the tuples nested or not? Perhaps this requires some fitness combiner which just takes an arbitrary method which the user defines based on what the wrapped fitnesses are.