[Feature Request] autotune metric minimize validation loss

[dhpollack]

It would be nice if one of the allowable metrics for autotune mode was minimizing the loss on the validation set. This would allow one to use autotune to create unsupervised vectors. Currently it's unclear what happens when one uses the autotune functionality when creating unsupervised vectors. It seems to run, but the Best Score is always reported as nan. If unsupervised autotune is something that is not implemented, there should be an error message rather than trying to do it, but always failing.

[Celebio]

Hi @dhpollack , The loss objective is something we were considering. Obtaining the loss from a specific set of inputs requires the rewrite of some consequent portion of the code. So we will probably implement this objective for autotune once this refactoring is done (unfortunately we can't commit to a date for that). Thank you for your suggestion!

And indeed, the autotune doesn't work with the unsupervised case. We will try to implement an appropriate error message for it.

Thanks again! Onur

[jyayoub]

hi, I have a problem with the autotune validation. I get this error Unknown argument: -autotune-validation any help on how to solve it ?

[ethanjperez]

Hi @Celebio, any update on support for using validation loss for autotune? If I use autotune to select a model based on F1, I end up with models that have a test loss that is worse than the uniform distribution on some datasets.