Metric not found in index

[PurenBITeam]

Hello, great package, works very well and helps a lot.

I only got stuck with one issue. Whenever I want to try another reduction technique (e.g. correlation or trees...) and define the reduction metric as "mae" or "mse" or "rmse", I get the message above.

What does this exactly mean?

thanks and br

christoph

[github-actions[bot]]

Welcome to Talos community! Thanks so much for creating your first issue :)

[mikkokotila]

Great to hear you find Talos useful :)

You have to add those metrics into your input model before you can use it for reduction. So if you do model.compile(...metrcs=['mae']...) into your input model, then you can use it in reduction.

[PurenBITeam]

This eas the case in my code...see below:

p = {'optimizer': [Adam, Adamax, Nadam, Adagrad, Adadelta], 'first_neuron': [5120, 1024, 512, 128, 64, 32, 16], 'batch_size': [100, 1000, 5000, 10000], 'epochs': [25], 'hidden_layers':[0, 1, 2, 3, 4], 'kernel_initializer': ['uniform', 'normal', 'glorot_uniform'], 'dropout': [0.0, 0.25, 0.5, 0.65, 2, 3, 4, 5, 10], 'losses': ["mape", "mse"], 'shapes': ['brick', 'triangle', 'funnel'], 'activation': ['relu'], 'lr': [0.0001, 0.0002, 0.0005, 0.0009, 0.001, 0.002, 0.005, 0.009, 0.01, 0.02, 0.035, 0.05, 0.075, 0.09, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] }

def fcmodel(X_train, y_train, x_val, y_val, params):

model = Sequential()

esr = 3
callbacks = [EarlyStopping(monitor="val_mean_absolute_error", patience=esr, min_delta=0.0001, mode="min",
                           baseline=0.1)]

model.add(Dense(params['first_neuron'], input_dim=X_train.shape[1],
                activation="relu",
                kernel_initializer = params['kernel_initializer'] ))
model.add(Dropout(params['dropout']))

## hidden layers
hidden_layers(model, params, 1)

model.add(Dense(1, activation="sigmoid",
                kernel_initializer=params['kernel_initializer']))

model.compile(loss=params["losses"], 
              optimizer=params['optimizer'](lr_normalizer(params['lr'], params['optimizer'])),
              metrics=['mae'])

out = model.fit(X_train, y_train, 
                    validation_data=[x_val, y_val],
                    batch_size=params['batch_size'],
                    epochs=params['epochs'],
                    callbacks=callbacks,
                    verbose=2)
return out, model

t = talos.Scan(x=X_df_train2.values, y=y_df_train2.values, model=fcmodel, params=p, x_val=X_df_val.values, y_val=y_df_val.values, seed=baseseed, experiment_name="TalosPipeline2_Corr", print_params=True, round_limit=250, reduction_method="correlation", reduction_interval=1, reduction_window=1, reduction_threshold=0.2, reduction_metric='mae', minimize_loss=True)

I still get the error message below:

---

KeyError Traceback (most recent call last)

in () 99 reduction_threshold=0.2, 100 reduction_metric='mae', --> 101 minimize_loss=True) 102 /data/anaconda/envs/py35/lib/python3.5/site-packages/talos/scan/Scan.py in __init__(self, x, y, params, model, experiment_name, x_val, y_val, val_split, random_method, seed, performance_target, fraction_limit, round_limit, time_limit, boolean_limit, reduction_method, reduction_interval, reduction_window, reduction_threshold, reduction_metric, minimize_loss, disable_progress_bar, print_params, clear_session, save_weights) 194 # input parameters section ends 195 --> 196 self._runtime() 197 198 def _runtime(self): /data/anaconda/envs/py35/lib/python3.5/site-packages/talos/scan/Scan.py in _runtime(self) 199 200 from .scan_run import scan_run --> 201 self = scan_run(self) /data/anaconda/envs/py35/lib/python3.5/site-packages/talos/scan/scan_run.py in scan_run(self) 24 # otherwise proceed with next permutation 25 from .scan_round import scan_round ---> 26 self = scan_round(self) 27 self.pbar.update(1) 28 /data/anaconda/envs/py35/lib/python3.5/site-packages/talos/scan/scan_round.py in scan_round(self) 26 # apply reductions 27 from ..reducers.reduce_run import reduce_run ---> 28 self = reduce_run(self) 29 30 try: /data/anaconda/envs/py35/lib/python3.5/site-packages/talos/reducers/reduce_run.py in reduce_run(self) 59 # check if correlation reducer can do something 60 if self.reduction_method == 'correlation': ---> 61 self = correlation(self, 'spearman') 62 63 # check if random forrest can do something /data/anaconda/envs/py35/lib/python3.5/site-packages/talos/reducers/correlation.py in correlation(self, method) 18 # transform the data properly first 19 from .reduce_utils import cols_to_multilabel ---> 20 data = cols_to_multilabel(self) 21 22 # get the correlations /data/anaconda/envs/py35/lib/python3.5/site-packages/talos/reducers/reduce_utils.py in cols_to_multilabel(self) 15 16 # drop all other metric columns except reduction_metric ---> 17 data = data[[self.reduction_metric] + self._param_dict_keys] 18 19 # convert all hyperparameter columns to multi label columns /data/anaconda/envs/py35/lib/python3.5/site-packages/pandas/core/frame.py in __getitem__(self, key) 2984 if is_iterator(key): 2985 key = list(key) -> 2986 indexer = self.loc._convert_to_indexer(key, axis=1, raise_missing=True) 2987 2988 # take() does not accept boolean indexers /data/anaconda/envs/py35/lib/python3.5/site-packages/pandas/core/indexing.py in _convert_to_indexer(self, obj, axis, is_setter, raise_missing) 1283 # When setting, missing keys are not allowed, even with .loc: 1284 kwargs = {"raise_missing": True if is_setter else raise_missing} -> 1285 return self._get_listlike_indexer(obj, axis, **kwargs)[1] 1286 else: 1287 try: /data/anaconda/envs/py35/lib/python3.5/site-packages/pandas/core/indexing.py in _get_listlike_indexer(self, key, axis, raise_missing) 1090 1091 self._validate_read_indexer( -> 1092 keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing 1093 ) 1094 return keyarr, indexer /data/anaconda/envs/py35/lib/python3.5/site-packages/pandas/core/indexing.py in _validate_read_indexer(self, key, indexer, axis, raise_missing) 1183 if not (self.name == "loc" and not raise_missing): 1184 not_found = list(set(key) - set(ax)) -> 1185 raise KeyError("{} not in index".format(not_found)) 1186 1187 # we skip the warning on Categorical/Interval KeyError: "['mae'] not in index" What did I do wrong? thanks and br

[mikkokotila]

It might be that your results will not have 'mae' because that's a shorthand in Keras, but it will be 'mean_average_error' instead.

As part of the troubleshoot, you can also try removing all the reduction related arguments from Scan():

reduction_method="correlation",
reduction_interval=1,
reduction_window=1,
reduction_threshold=0.2,
reduction_metric='mae'

[PurenBITeam]

Thanks...It's working now.