autonomio / talos

Hyperparameter Experiments with TensorFlow and Keras
https://autonom.io
MIT License
1.62k stars 269 forks source link

Reshape input data when evaluating models based on conv1d #387

Closed beeb closed 1 year ago

beeb commented 4 years ago

When using the scan_object.evaluate_models() together with AutoModel, the models generated with the type "conv1d" expect a reshaped input.

AutoModel does the reshaping during training with wrangle.array_reshape_conv1d but this transformation is not performed before evaluation.

I think if people are gonna use evaluate_models, this should be checked since the input layers can expect different shapes depending on the AutoScan results.

mikkokotila commented 4 years ago

Good catch. I guess the simplest way to do this would be to check the weight shape in evaluate (which is in any case useful for integrity) and the throw an error if the shape is wrong and say what shape it would be. I'll have to check carefully to be sure, but I think right now there is some inconsistency in terms of taking the shape of input data depending on if its 2d or not, which can be dealt with as part of this.

mikkokotila commented 4 years ago

Related with this issue, what do you think should be the behavior? I think it's generally good to avoid "magics" unless it is under autom8 which is all about it. Seems like there are couple of options:

a) leave it as it is but have a custom error message that shows what the shape of the data is vs. what it should be (but I think this is what's already happening with Keras) b) leave it as it but instruct exactly how to reshape the data (this is tricker as the same case might appear in non autom8 uses as well) c) figure out a way where whatever data transformation was done in the experiment, is done again d) something else

What do you think?