Run Hyperas with Siamese network

[haramoz]

I need to run the hyperas for optimizing my siamese network, But I am unable to do so. Following is my set up codes, and the error. BRANCHES OF SIAMESE:

def create_base_network(input_shape): random_seed = 7

model = Input(shape=input_shape)
x = Conv2D(16, (3, 3), padding="same",activation='relu', data_format='channels_first',
           kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
          bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(model)

x = Conv2D(16, (3, 3), padding="same", activation='relu',
           kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
          bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)
x = MaxPooling2D(pool_size=(2, 2),strides=(2, 2))(x)

x = Conv2D(32, (3, 3), padding="same", activation='relu',
           kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
          bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)

x = Conv2D(32, (3, 3), padding="same", activation='relu',
           kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
          bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)

x = MaxPooling2D(pool_size=(2, 2),strides=(2, 2))(x)
x = Flatten()(x)
x = Dense({{choice([512,96,1024,2048,4096])}}, kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)
Dropout({{uniform(0, 1)}})
x = Dense({{choice([512,96,1024,2048,4096])}}, kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)
Dropout({{uniform(0, 1)}})
x = Dense({{choice([512,96,1024,2048,4096])}}, kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)

return Model(model,x)

HOW I CREATE THE MODEL def create_model(x_train1, x_train2, y_train,x_val1, x_val2, y_val, x_test1, x_test2, y_test): epochs = 30 inputshape = (1,96,96) patience = 1

base_network = create_base_network(input_shape)

input_a = Input(shape=input_shape)
input_b = Input(shape=input_shape)

processed_a = base_network(input_a)
processed_b = base_network(input_b)

opt = Adam(lr={{choice([1e-3,1e-4,1e-5])}})
distance = Lambda(euclidean_distance,
                  output_shape=eucl_dist_output_shape)([processed_a, processed_b])
model = Model([input_a, input_b], distance)
model.compile(loss=contrastive_loss_altered, optimizer=opt, metrics=["accuracy"])

es = EarlyStopping(monitor='val_acc', patience=patience_,verbose=1)
checkpointer = ModelCheckpoint(filepath='keras_weights.hdf5',
                               verbose=1,
                               save_best_only=True)
model.fit([x_train1, x_train2],y_train,
      batch_size={{choice([32,64,128])}},
      epochs=epochs,
      validation_data=([x_val1,x_val2], y_val),
      callbacks=[es],
      verbose=0)

score, acc = model.evaluate([x_test1, x_test2], y_test)

MAIN:

if name == 'main':

x_train1, x_train2, y_train,x_val1, x_val2, y_val, x_test1, x_test2, y_test = data()

best_run, best_model = optim.minimize(model=create_model, data=data,
functions = [process_data,create_base_network,euclidean_distance,eucl_dist_output_shape,contrastive_loss_alter$
algo=tpe.suggest,max_evals=1,trials=Trials())

THEN THE ERROR I AM GETTING: Error:

Using TensorFlow backend. Traceback (most recent call last): File "hyperas_contrastive_loss.py", line 152, in algo=tpe.suggest,max_evals=1,trials=Trials()) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperas/optim.py", line 67, in minimize verbose=verbose) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperas/optim.py", line 133, in base_minimizer return_argmin=True), File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 307, in fmin return_argmin=return_argmin, File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/base.py", line 635, in fmin return_argmin=return_argmin) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 320, in fmin rval.exhaust() File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 199, in exhaust self.run(self.max_evals - n_done, block_until_done=self.async) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 173, in run self.serial_evaluate() File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/fmin.py", line 92, in serial_evaluate result = self.domain.evaluate(spec, ctrl) File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/base.py", line 840, in evaluate rval = self.fn(pyll_rval) File "/home/amalli2s/thesis/keras/temp_model.py", line 222, in keras_fmin_fnct File "/home/amalli2s/thesis/keras/temp_model.py", line 192, in create_base_network File "/home/amalli2s/anaconda3/envs/iwin/lib/python3.6/site-packages/hyperopt/pyll_utils.py", line 21, in wrapper raise TypeError('require string label') TypeError: require string label

This error seems to be popular, but i don't see a specific solution or guide line to fix it yet. Can any one help me with this issue?

[maxpumperla]

@haramoz can you share your full example? I need to see how you define create_model and data functions in detail.

[maxpumperla]

this error continues to be "popular" despite all efforts at pointing to the examples, most closed issues point to this reference: https://github.com/maxpumperla/hyperas/issues/106#issuecomment-339894530

check your example syntactically against examples and tests, please.

[haramoz]

Hello,

So below are the complete versions of the data functions, process data is loading data from the pickle file. and data function is just calling it.

def process_data():
    filename = 'sonar_data'
    infile = open(filename,'rb')
    sonar_data_dict = pickle.load(infile)

    infile.close()
    x_train1 = sonar_data_dict.get("x_train1")
    x_train2 = sonar_data_dict.get("x_train2")
    y_train = sonar_data_dict.get("y_train")

    x_val1 = sonar_data_dict.get("x_val1")
    x_val2 = sonar_data_dict.get("x_val2")
    y_val = sonar_data_dict.get("y_val")

    x_test1 = sonar_data_dict.get("x_test1")
    x_test2 = sonar_data_dict.get("x_test2")
    y_test = sonar_data_dict.get("y_test")

    y_test_inverted = 1 - y_test
    y_train_inverted = 1 - y_train
    y_val_inverted = 1 - y_val

    return x_train1, x_train2, y_train_inverted, x_val1, x_val2, y_val_inverted, x_test1, x_test2, y_test_inverted

def data():
    x_train1, x_train2, y_train, x_val1, x_val2, y_val, x_test1, x_test2, y_test = process_data()
    return x_train1, x_train2, y_train, x_val1, x_val2, y_val, x_test1, x_test2, y_test

`def create_base_network(input_shape):
    random_seed = 7

    model = Input(shape=input_shape)
    x = Conv2D(16, (3, 3), padding="same",activation='relu', data_format='channels_first',
               kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
              bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(model)

    x = Conv2D(16, (3, 3), padding="same", activation='relu',
               kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
              bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)
    x = MaxPooling2D(pool_size=(2, 2),strides=(2, 2))(x)

    x = Conv2D(32, (3, 3), padding="same", activation='relu',
               kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
              bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)

    x = Conv2D(32, (3, 3), padding="same", activation='relu',
               kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
              bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)

    x = MaxPooling2D(pool_size=(2, 2),strides=(2, 2))(x)
    x = Flatten()(x)
    x = Dense(512, kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)
    Dropout({{uniform(0, 1)}})
    x = Dense(512, kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)
    Dropout({{uniform(0, 1)}})
    x = Dense(512, kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)

    return Model(model,x)

def create_model(x_train1, x_train2, y_train,x_val1, x_val2, y_val, x_test1, x_test2, y_test):
    epochs = 30
    input_shape = (1,96,96)
    patience_ = 1

    base_network = create_base_network(input_shape)

    input_a = Input(shape=input_shape)
    input_b = Input(shape=input_shape)

    processed_a = base_network(input_a)
    processed_b = base_network(input_b)

    opt = Adam(lr={{choice([1e-3,1e-4,1e-5])}})
    distance = Lambda(euclidean_distance,
                      output_shape=eucl_dist_output_shape)([processed_a, processed_b])
    model = Model([input_a, input_b], distance)
    model.compile(loss=contrastive_loss_altered, optimizer=opt, metrics=["accuracy"])

    es = EarlyStopping(monitor='val_acc', patience=patience_,verbose=1)
    checkpointer = ModelCheckpoint(filepath='keras_weights.hdf5',
                                   verbose=1,
                                   save_best_only=True)
    model.fit([x_train1, x_train2],y_train,
          batch_size={{choice([32,64,128])}},
          epochs=epochs,
          validation_data=([x_val1,x_val2], y_val),
          callbacks=[es],
          verbose=0)

    score, acc = model.evaluate([x_test1, x_test2], y_test)
    print('Test accuracy:', acc)
    return {'loss': -acc, 'status': STATUS_OK, 'model': model}

Ok so this is what i have right now. I followed the other threads and your examples. One thing I have observed is that, I can SUCCESSFULLY use hyperas(CHOICE) in my create_model function. I tried batchsize, learning rate, decay only and they worked. When I apply choice inside the create_base_network, like applying uniform to the Dropouts or choice to the dense layers. It does not work. I shared the base model again. For my base network, I am just following this keras, example

[haramoz]

I have compared the syntax with the examples, not seeing any issue, I have seen the comment on issue #106. It does not add any value in my case, hence thought of opening a new issue, may be siamese is somehow messing up the working or so by calling same network branch twice or something is my thought process behind this. Thank you for your prompt response.

[maxpumperla]

@haramoz I think you answered your own question. hyperas can't handle auxiliary functions out of the box. it only parses data and model functions, nothing outside that scope. see this for how to solve that:

https://github.com/maxpumperla/hyperas/blob/master/examples/use_intermediate_functions.py#L85-L88

please close this issue if it indeed fixes your problem

[haramoz]

best_run, best_model = optim.minimize(model=create_model, data=data,
    functions = [process_data,create_base_network,euclidean_distance,eucl_dist_output_shape,contrastive_loss_altered],
    algo=tpe.suggest,max_evals=1,trials=Trials())

You see I am supplying the create_base_network function as the intermediate function following that example only. Without it, the function name is not identified. But just passing the name of an intermediate function does not make it eligible for using CHOICE or UNIFORM in it? as the intermediate functions are not participating in the search space? If that is true then my code will never work. And its worth closing the issue, since it's about a feature which is not there yet. I will wait for your confirmation once on this.

If there is any confusion about my above question, from the example that u shared. Can I use CHOICE/UNIFORM in the visualization_mnist intermediate function ??

[maxpumperla]

sorry, but hyperas was not intended to be used that way. only the model function is searched for handles.

shall we close this and you please create a feature request to support this?

[maxpumperla]

and you're right that it's not clear that this is not possible

[haramoz]

I am satisfied with this conversation and have clarity now. Thank you very much. Wish you the best. :)

[haramoz]

Hello again,

I think I was able to idea for of an work around of this problem. From your explanation it was clear that the "magic" happens only in the function passed as the model and not to the intermediate functions. So from this I had the idea that if i create the search space(choice[1,2,3]) inside the model function and then pass the variable/placeholder to the intermediate function it should work with the current logic. And it seems it might!! Now the created space:

return { **'dense_filter': hp.choice('dense_filter', [512,1024]),** 'decay': hp.choice('decay', [1e-2,1e-3,1e-4,1e-5,1e-6,1e-7]), } Resulting replaced keras model:

1: def keras_fmin_fnct(space):
   2: 
   3:     epochs = 30
   4:     input_shape = (1,96,96)
   5:     patience_ = 1
   6:     **dense_filter = space['dense_filter']**
   7:     base_network = create_base_network(input_shape,dense_filter)
   8: 
   9:     input_a = Input(shape=input_shape)
  10:     input_b = Input(shape=input_shape)
  11: 
  12:     processed_a = base_network(input_a)
  13:     processed_b = base_network(input_b)
  14: 
  15:     opt = Adam(lr=1e-5,decay=space['decay'])
  16:     distance = Lambda(euclidean_distance,
  17:                       output_shape=eucl_dist_output_shape)([processed_a, processed_b])
  18:     model = Model([input_a, input_b], distance)
  19:     model.compile(loss=contrastive_loss_altered, optimizer=opt, metrics=["accuracy"])

The replaced keras model looks like this. While the intermediate function is as follows:

` def create_base_network(input_shape,dense_filter): random_seed = 7

model = Input(shape=input_shape)
x = Conv2D(16, (3, 3), padding="same",activation='relu', data_format='channels_first',
           kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
          bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(model)

x = Conv2D(16, (3, 3), padding="same", activation='relu',
           kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
          bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)
x = MaxPooling2D(pool_size=(2, 2),strides=(2, 2))(x)

x = Conv2D(32, (3, 3), padding="same", activation='relu',
           kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
          bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)

x = Conv2D(32, (3, 3), padding="same", activation='relu',
           kernel_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed),
          bias_initializer=RandomNormal(mean=0.0, stddev=0.05, seed=random_seed))(x)

x = MaxPooling2D(pool_size=(2, 2),strides=(2, 2))(x)
x = Flatten()(x)
x = Dense(dense_filter, 
kernel_initializer=keras.initializers.he_normal(seed=random_seed),activation='relu')(x)

return Model(model,x)

` My very first run after the finding is still running. Do you think this is actually might work? I will get back to you on this.

[maxpumperla]

cool, keep me posted about this

[haramoz]

I believe I have good news for us. It works! The model predicts the best combination like this. best run {'decay': 1, 'dense_filter': 0} Here the dense_filter was passed to the intermediate function from the create_model function. So that is one way of doing it seems. Do you think i should send you an example of this, so that others, if needed can look it up? Please let me know. Thanks for your time. :)

[maxpumperla]

yes, please! do you feel like you could do a pull request? otherwise send the example in some other form

[haramoz]

ok I will make a simpler easy to understand example and do a pull request tomorrow. After another round of testing of course. :)

[maxpumperla]

of course, no stress... and thank you. you could also add your case to existing unit tests, but an example is just fine. your call!

[haramoz]

ok Hi, I was able to create a pull request, where i placed the example in the example folder of my branch. Let me know your comments please. I ran it for 50 evals and then took the suggested params and was able to improve the results by little bit. Please let me know if this is ok. I do not have extra test cases yet.