Concatenate BERT and CNN

[efabb95]

I'm trying to concatenate two model in one. I have a BERT model and EfficientNet model.

input_text = model_Bert.inputs[:2]
text = model_Bert(input_text)
input_img = model_EfNet.layers[0].input
img = model_EfNet(input_img)
concatenated = layers.concatenate([text, img], axis=1) #same dimension
dense = layers.Dense(512, activation='relu')(concatenated)
dense = layers.Dense(128, activation='relu')(dense)
dense = layers.Dropout(0.3)(dense)
outputs = layers.Dense(2, activation='softmax', name = 'real_output')(dense)
model_Multimodal = keras.models.Model(inputs=[input_text, input_img], outputs=outputs)

But I got this error:

ValueError Traceback (most recent call last) in 9 outputs = layers.Dense(2, activation='softmax', name = 'real_output')(dense) 10 ---> 11 model_Multimodal = keras.models.Model(inputs=[input_text, input_img], outputs=outputs)

~/anaconda3/lib/python3.7/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs) 89 warnings.warn('Update your ' + object_name + ' call to the ' + 90 'Keras 2 API: ' + signature, stacklevel=2) ---> 91 return func(*args, **kwargs) 92 wrapper._original_function = func 93 return wrapper

~/anaconda3/lib/python3.7/site-packages/keras/engine/network.py in init(self, *args, **kwargs) 92 'inputs' in kwargs and 'outputs' in kwargs): 93 # Graph network ---> 94 self._init_graph_network(*args, **kwargs) 95 else: 96 # Subclassed network

~/anaconda3/lib/python3.7/site-packages/keras/engine/network.py in _init_graph_network(self, inputs, outputs, name, **kwargs) 167 'must come from keras.layers.Input. ' 168 'Received: ' + str(x) + --> 169 ' (missing previous layer metadata).') 170 # Check that x is an input tensor. 171 layer, node_index, tensor_index = x._keras_history

ValueError: Input tensors to a Model must come from `keras.layers.Input`. Received: [<tf.Tensor 'Input-Token_1:0' shape=(None, 128) dtype=float32>, <tf.Tensor 'Input-Segment_1:0' shape=(None, 128) dtype=float32>] (missing previous layer metadata).

How can I merge this two models?

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[mathshangw]

excuse me did you solve it ?

[efabb95]

Hi @mathshangw, I took embeddings of this two models and then fuse them in a new one network, but performance was not very satisfying. This is my thesis, if you want to check my solution you can find the code here: https://github.com/efabb95/Emotion-recognition-with-neural-network-attention-based-model

[mathshangw]

thanks a lot for replying and the link .. excuse me I'm trying to concatenate two models by extracting the embedding for the sentence but got ValueError: Layer weight shape (30522, 768) not compatible with provided weight shape torch.Size([1, 15, 768])

[efabb95]

I didn't use pytorch, however you have to check your input because you are passing a matrix instead of a tensor.

[mathshangw]

thanks a lot, another question, please . I read that bert work on subwords so if I need to get the caption for the image , should i use sentence embedding or word embedding ? appreciate your time

[efabb95]

Do you carry out hate speech recognition/emotion recognition or something else? BERT returns in the first position [0] the sentence embeddings, in the other positions [1..N] the words embeddings. For me it's better to use sentence embeddings so that you don't have to deal with the different number of words in each sentence. In my thesis I used only sentence embeddings.