Closed bjtho08 closed 5 years ago
Can you try the v.0.6 version:
pip install git+http://github.com/autonomio/talos@daily-dev
The logging aspect is entirely rebuilt in it, and these kinds of issues should no longer appear.
@mikkokotila I am already running 0.6 which I installed after the latest merge from params-api. Did you make significant changes to the logging in the last month or so?
I did not. Thanks for clarification. I'll try to look into this tomorrow.
In the meantime, can you share your experiment log? Or the first rows of it.
Sorry, I am not sure what you are referring to. Do you mean the output during scanning?
On the machine where you are working, there is a a .csv file being updated on each permutation throughout the experiment, which is identical to what is being stored in the end to the scan_object
. That one.
Here it is (i renamed it .txt because github complained about the .csv format) 062519213324.csv.txt
Looking at the log, it does seem like some weird stuff is going on in the first and last columns. The last column looks like it should have been 11 individual columns, based on the contents of the bottom row, but I don't see why it would think so. Is it related to how I pass static and dynamic parameters into the model? I made a nested function for my model to handle some technical difficulties related to generators and in the beginning of the inner function, I update the params dictionary with all the static parameters that I pass to the outer function, such that I can freely switch back and forth between having the parameters be part of the parameter space and not take up unnecessary in the log file or clutter the correlation matrix.
I've taken the time to look closer at the log file and it is indeed a concatenation of the static and grid params, but there is something odd about the header column. One of the grid param keys is missing from the header, which means that even if I would remove the concatenation step, it would still have resulted in an error. How is the header column generated? I would have assumed that it reads the param.keys()
, but then the missing header ("class_weights"
) should not be possible. I Hope you have enough information to find a solution. I am thinking of creating an internal dict in the inner function and updating it with both the params
and static_params
dicts to preserve the original params
dict.
Just to be safe, can you do:
import talos
talos.__version__
Also, can you run a minimal example for a sanity check:
import talos as ta
from keras.models import Sequential
from keras.layers import Dense
def minimal():
x, y = ta.templates.datasets.iris()
p = {'activation':['relu', 'elu'],
'optimizer': ['Nadam', 'Adam'],
'losses': ['logcosh'],
'hidden_layers':[0, 1, 2],
'batch_size': [20,30,40],
'epochs': [10,20]}
def iris_model(x_train, y_train, x_val, y_val, params):
model = Sequential()
model.add(Dense(32, input_dim=4, activation=params['activation']))
model.add(Dense(3, activation='softmax'))
model.compile(optimizer=params['optimizer'], loss=params['losses'])
out = model.fit(x_train, y_train,
batch_size=params['batch_size'],
epochs=params['epochs'],
validation_data=[x_val, y_val],
verbose=0)
return out, model
scan_object = ta.Scan(x, y, model=iris_model, params=p, fraction_limit=0.1)
return scan_object
minimal()
Let's see if the same issue persists.
I can see that the parameter dictionary you are posting above does not match the model or the expected output you mention. Is it that you add parameters after starting the experiment? The headers columns are based on reading the parameter dictionary at the beginning of the experiment. So that would cause a problem.
Here is the requested version output:
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import talos
/home/bjarne/.pyenv/versions/anaconda3-5.2.0/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
>>> talos.__version__
'0.6.0'
I am ssh'ing from my phone, so the sanity check will have to wait until later.
Regarding the parameters, as I mentioned, I am combining two dictionaries inside the model function in order to minimize rewriting code when I want to change parameters to test while also keeping the number of tracked parameters to a minimum, in order to avoid cluttering the correlation matrix with static parameters.
I ran your MWE and it completed without error. The output csv is here: 062719235236.csv.txt
Regarding the parameters, as I mentioned, I am combining two dictionaries inside the model function in order to minimize rewriting code when I want to change parameters to test while also keeping the number of tracked parameters to a minimum, in order to avoid cluttering the correlation matrix with static parameters.
That will be the cause of this. What we do is create a record for the parameters at the beginning of the scan, and then universally use that throughout the pipeline to ensure consistency. If you load all your parameters in the params dictionary as intended use assumes, you will no longer suffer from this.
It seems that you have a justification for doing things otherwise, so I think we can explore that as a feature request. The best way to do that would be to create a new ticket for that specific request, and provide an example where you use the minimal()
from above, but altered in a way that makes your case blatantly clear.
Regarding this ticket, I'll leave it open just in case for a couple of days, but it seems that the issue itself is resolved by a) using the intended approach b) escalating to a feature requested the new use.
Thanks a lot for working together to figure this out. Have a nice day ahead too :)
I decided to rewrite the function such that all parameters are copied to an internalized dictionary at the very start of the inner function which avoids tampering with the params
dict fed by talos.Scan
. This fixed the issue. For reference, the relevant changes are:
def talos_presets(weight_path, cls_wgts, static_params, train_generator, val_generator):
...
def talos_model(x, y, val_x, val_y, params):
...
# Dummy inputs
_ = x, y, val_x, val_y
internal_params = OrderedDict()
internal_params.update(static_params)
internal_params.update(params)
...
return history, model
return talos_model
Thanks for helping me with the bug hunt! It's nice to work with committed devs! Have a great day :)
Thanks to you too :) I'm glad you got it sorted out. And likewise, have a nice day!
I ran into this issue today. I was passing _inputshape and _numclasses into my params dict (with the goal of consolidating all model-related variables in one place), which was the apparent cause of the issue. After removing these variables from the params dict, the error went away.
Bug description
Talos fails upon completion of all rounds in parameter search because of a mismatch between axis and new values.
Output of shape for x and y
X.shape = (16, 208, 208, 3) Y.shape = (16, 208, 208, 10)
Talos params dictionary
The Keras model wired for Talos
Traceback