Closed beeb closed 5 years ago
Totally agree with you that this is done very poorly. Can you check the upcoming v.0.6 version, the whole thing, as well logging more broadly is now handled in a reasonable way. Given that you know what you are doing, I'd recommend to move into that version.
Basically this will be the first release of the next LTS. The current pip version v.0.4.9 will likely remain as it is, given that for a couple of months there is barely any new bugs and the old ones are handled to the extent that relate with common use-cases.
Hey, thanks for the reply! I'll have a look into 0.6, nice to see that it's still a very much active project! I won't be able to actively use it for now because the code is used in production, but I'll dig into it if I have some spare time. Any chance to see a documentation draft with the main changes in the near future?
Given that most of the recent tickets have to do with documentation, I think it can't be avoided. This has to be done before v0.6 becomes the new master. I'm committed to creating an all-encompassing documentation, and then keep it up-to-date with all future updates.
Closing here. Feel free to open new ticket if anything.
The current code will select the best epoch by looking at the maximum value for metrics containing the text
acc
and default for everything else to taking the minimum value as criterion. https://github.com/autonomio/talos/blob/6a4fbfacdbd7a6ebfddd27668761089978cfc053/talos/utils/results.py#L45I propose to change this behavior to have the code look if the text
loss
is present in the key, in which case the minimum value is taken, and to default to maximum value for everything else. Anything that isn't defined as a loss should be a performance metric where in a majority of cases higher values mean it's better.I'm open to discussion! A best solutions would be to be able to define this on a per-key basis to avoid ambiguity but I don't see this happening easily without changing the code base some more.