Open eschibli opened 3 years ago
As a temporary solution, you may try using random search as the tuning algorithm. We will fix this issue systematically later.
Thanks for the suggestion haifeng, that should help in the meantime.
Hi @eschibli! Did you try tuner = 'random'?. I am trying something similar to your test but I have the same problem. I am further limiting the model size to only 153600 parameters in order to find small architectures for tensorflow lite for microcontrollers. Could you suggest me some way to use ImageClassifier for this? Thank you so much!
@haifeng-jin @elloza
I did try, and as elloza says, it isn't working. Varying the tuner does not appear to prevent the same model from being repeatedly tried and rejected for size.
I meet the same problem
I somehow realized that the default (greedy) tuner would be more suitable to solve this problem. It would go through a list of models ( I expect at least one of those would work and not over-sized), and make modifications to that model.
From the error message above, it seems there is a bug. It fails while going through the initial model list. I will inspect it.
A possible fix to pull request:
@haifeng-jin Without looking at the current code, it seems like simply assigning very high or Inf
loss to invalid models would solve the problem rather elegently.
any updates to that topic? I'm also interested in small models with max. 150.000 parameters.
Are the better solutions than changing the self._max_fail_streak in kerastuner/engine/hypermodel.py so far?
This appears to be fixed in v1.1 for max_model_size>=23555082
(the size of the hardcoded Resnet model?). Anything smaller and a FailedTrialError
is raised and not handled.
Bug Description
When
max_model_size
is set below 63812570, ImageClassifier will repeatedly try to build a model with an EfficientNet-B7 backbone, then throw a RuntimeError. EfficientNet-B7 is a very large model - too large to manage with most GPUs - so it would certainly be nice to be able to easily define a smaller maximum number of model weights.I know it is possible to work around this issue by defining a custom search space that excludes the largest backbones, but it does not appear to be simple to exclude all of the large ones while keeping, say, the smaller versions of EfficientNet.
Bug Reproduction
Code:
Output:
Expected Behavior
ImageClassifier does not repeatedly try backbones with more weights than
max_model_size
and explores the remaining parameter space.Setup Details
Additional context
I am new to automl, but not to machine learning, so please bear with me if I'm missing something obvious. I searched for max-model_size and didn't see another post about this issue.