autonomio / talos

Hyperparameter Experiments with TensorFlow and Keras
https://autonom.io
MIT License
1.62k stars 268 forks source link

[FEATURE REQUEST] a key-blacklist or -whitelist as Scan parameter #272

Closed tik0 closed 2 years ago

tik0 commented 5 years ago

It is currently not possible to add user defined obejcts (like numpy arrays or other objects) to the keras history object in a scan loop. The result evaluations in results.py (e.g. 50 and 55 make comparisions of the values, which results in errors for objects like np.array. However, a simple workaround is the scan option ta.Scan(..., last_epoch_value=True, ...) which disables the evaluations. But since this is not a desired behaviour, I would propose a key-blacklist or -whitelist as Scan parameter, so that the user can define which keys are evaluated or not.

tik0 commented 5 years ago

So I elaborated a bit on that and I think there is even a small issue with ta.Scan(..., last_epoch_value=True, ...). Whether or not last_epoch_value is True or False, the peak epochs will be evaluated. This causes definitely errors for user-defined history objects. Making the peak epoch evaluation depending on last_epoch_value, which is absolutely plausible if the user only demands that behavior, fixes the issue. Here is a working commit in my talos-fork: https://github.com/tik0/talos/commit/dd3cebf4ed40f8965372bba0a14eaa54178bc8fd

If a patch is desired, I will elaborate on a PR!

mikkokotila commented 5 years ago

Do I understand correct, you would like to add things to the history object, which are not processed by Talos? Can you give an example of where you might want to do that.

To avoid doubt last_epoch_value=True means that what is reported in Talos results from Scan() is the result of the last epoch, vs. last_epoch_value=False means that peak epoch value is recorded instead.