Merge two models with multiple outputs

[boyander]

Hi! I am trying to merge two models to train my deepnet. I'm using a function as the merge mode as defined in the API, but keras gave me the following error: Exception: Output tensors to a Model must be Keras tensors. Found: Join.0

Here's the code i'm using:

def dotpMerge(inputs):
    plane_sweep_color = inputs[0]
    depth_probabilities = inputs[1]

    r_map = []
    g_map = []
    b_map = []
    for i in range(0, len(plane_sweep_color)):
        plane = plane_sweep_color[i]
        # Multiply R,G,B
        r_map.append(K.dot(plane[0], depth_probabilities[0]))
        g_map.append(K.dot(plane[1], depth_probabilities[1]))
        b_map.append(K.dot(plane[2], depth_probabilities[2]))

    final_r = r_map[0]
    final_g = g_map[0]
    final_b = b_map[0]
    for i in range(1, len(r_map)):
        final_r += r_map[i]
        final_g += g_map[i]
        final_b += g_map[i]

    return K.concatenate([final_r, final_g, final_b])

# Output dot product
color_model = Model(input=color_inputs, output=color_outputs)
select_model = Model(input=select_inputs, output=select_outputs)
output_towers_merge = merge([color_outputs, select_outputs], output_shape=(None, 8, 8, 3), mode=dotpMerge)

# Create the model
model = Model(color_inputs + select_inputs, output_towers_merge)
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

Any ideas on what i'm doing wrong? is there any issue with keras related to this? Thanks in advance.

[boyander]

I've forgot to mention that color_model have multiple output, so in fact this is not a normal merge, because one of the models have N ouputs and the other one has 1 output. How to merge those is my question.

[fchollet]

Your function does not output a Keras tensor, but rather a Theano tensor, which cannot be used to define a Model. To output a Keras tensor, wrap your function into a Lambda layer: http://keras.io/layers/core/#lambda

[boyander]

@fchollet It is not working. The same error "Found: Join.0" is given. How should i proceed on that?

Thanks!

output_towers_merge = merge([color_outputs, select_outputs], output_shape=(None, 8, 8, 3), mode=dotpMerge)
lam = Lambda(lambda x: x, output_shape=(8, 8, 3))
lam.build((8, 3, 3))
out = lam(output_towers_merge)

# Create the model
model = Model(input=color_inputs + select_inputs, output=out)

[fchollet]

Lambda(dotpMerge, output_shape=...)

[boyander]

@fchollet Same thing Output tensors to a Model must be Keras tensors. Found: Join.0. Don't know where i'm wrong.

lam = Lambda(dotpMerge, output_shape=(8, 8, 3))
lam.build((8, 3, 3))
out = lam([color_outputs, select_outputs])

# Create the model
model = Model(input=color_inputs + select_inputs, output=out)

[fchollet]

Please read the documentation.

On 2 July 2016 at 13:11, Marc Pomar Torres notifications@github.com wrote:

@fchollet https://github.com/fchollet Same thing Output tensors to a Model must be Keras tensors. Found: Join.0. Don't know where i'm wrong.

lam = Lambda(dotpMerge, output_shape=(8, 8, 3)) lam.build((8, 3, 3)) out = lam([color_outputs, select_outputs])

Create the model

model = Model(input=color_inputs + select_inputs, output=out)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fchollet/keras/issues/3130#issuecomment-230119836, or mute the thread https://github.com/notifications/unsubscribe/AArWb9GZx94tljzmVUiF9ATz_9qirN6mks5qRsYDgaJpZM4JDuXA .

[minogame]

I got the same problem,

x_u = Input(shape=(96,96,3))

f_u = shared_cnn(x_u)

y_u = do_something(f_u)

y_ut = Dense(output_dim=10)(y_u)

model_pred = Model(x_u, y_ut)

and it told me that

Output tensors to a Model must be Keras tensors. Found: Tensor("add_711:0", shape=(?, 10), dtype=float32)

so what you have to do is to manually add _keras_history and _keras_shape to the output Tensor, which sounds stupid but works.

[Lieutenant-Tom]

I have no idea on how to add _keras_history and _keras_shape to the output Tensor?

[DawnMe]

@Lieutenant-Tom Try with mymodel.layers[-1].output._keras_history='',mymodel.layers[-1].output._keras_shape=...

[TiRune]

Doing that just like that @DawnMe gives you

layer, node_index, tensor_index = x._keras_history ValueError: not enough values to unpack (expected 3, got 0)