search

autonomio / talos

Hyperparameter Experiments with TensorFlow and Keras
https://autonom.io
MIT License
1.62k stars 268 forks source link

[Improvement] selection of best epoch: loss vs metric #303

Closed beeb closed 5 years ago

beeb commented 5 years ago

The current code will select the best epoch by looking at the maximum value for metrics containing the text acc and default for everything else to taking the minimum value as criterion. https://github.com/autonomio/talos/blob/6a4fbfacdbd7a6ebfddd27668761089978cfc053/talos/utils/results.py#L45

I propose to change this behavior to have the code look if the text loss is present in the key, in which case the minimum value is taken, and to default to maximum value for everything else. Anything that isn't defined as a loss should be a performance metric where in a majority of cases higher values mean it's better.

I'm open to discussion! A best solutions would be to be able to define this on a per-key basis to avoid ambiguity but I don't see this happening easily without changing the code base some more.

mikkokotila commented 5 years ago

Totally agree with you that this is done very poorly. Can you check the upcoming v.0.6 version, the whole thing, as well logging more broadly is now handled in a reasonable way. Given that you know what you are doing, I'd recommend to move into that version.

Basically this will be the first release of the next LTS. The current pip version v.0.4.9 will likely remain as it is, given that for a couple of months there is barely any new bugs and the old ones are handled to the extent that relate with common use-cases.

beeb commented 5 years ago

Hey, thanks for the reply! I'll have a look into 0.6, nice to see that it's still a very much active project! I won't be able to actively use it for now because the code is used in production, but I'll dig into it if I have some spare time. Any chance to see a documentation draft with the main changes in the near future?

mikkokotila commented 5 years ago

Given that most of the recent tickets have to do with documentation, I think it can't be avoided. This has to be done before v0.6 becomes the new master. I'm committed to creating an all-encompassing documentation, and then keep it up-to-date with all future updates.

Closing here. Feel free to open new ticket if anything.