Call new layer on the last layer of create_model object using Functional API

[ztsv-av]

Hi. First, I want to say that I enjoy this library a lot! Thank you @martinsbruveris for creating it!

I have a question: I want create a model body using create_model function and add my own classification head. In classification head I want to add another input layer to additional features, call a concatenate layer on last layer of the create_model object and new input layer, and add final dense layer. Since create_model object is not a Sequential or Functional model object, is there any way I can do that? I tried using 'model_tfimm.output' or 'model_tfimm.layers[-1].output' calls, because .output call works with Tensorflow models, but it does not seem to work with tfimm models:

dense_1 = tf.keras.layers.Dense(512, activation='relu', name='dense_1')(model_tfimm.layers[-1].output)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
v:\Git\spellbook\magicClassification.py in <module>
----> 1 dense_1 = tf.keras.layers.Dense(512, activation='relu', name='dense_1')(model_tfimm.layers[-1].output)

~\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\engine\base_layer.py in output(self)
   2094     """
   2095     if not self._inbound_nodes:
-> 2096       raise AttributeError('Layer ' + self.name + ' has no inbound nodes.')
   2097     return self._get_node_attribute_at_index(0, 'output_tensors', 'output')
   2098 

AttributeError: Layer activation_72 has no inbound nodes.

model_tfimm.output

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
v:\Git\spellbook\magicClassification.py in <module>
----> 1 model_tfimm.output

~\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\engine\base_layer.py in output(self)
   2094     """
   2095     if not self._inbound_nodes:
-> 2096       raise AttributeError('Layer ' + self.name + ' has no inbound nodes.')
   2097     return self._get_node_attribute_at_index(0, 'output_tensors', 'output')
   2098 

AttributeError: Layer conv_ne_xt_1 has no inbound nodes.

Using Tensorflow Functional API this would like something like this:

model_tfimm = tfimm.create_model(TFIMM_MODEL_NAME, nb_classes=0, pretrained="timm")
feature_extractor = model_tfimm.output

add_input = tf.keras.layers.Input(shape=(NUM_ADD_FEATURES, ), name='input_features_layer')
concat_layer = tf.keras.layers.Concatenate(name='concat_features')([feature_extractor, add_input])

predictions = tf.keras.layers.Dense(NUM_CLASSES, activation=OUTPUT_ACTIVATION)(concat_layer)

model = tf.keras.Model(inputs=[model_tfimm.input, add_input], outputs=predictions)

Any ideas?

[ztsv-av]

I noticed that tfimm models do not have an Input layer. Is it even possible to train a custom model with tfimm body with multiple inputs? I used this code to build a model I wanted, not sure it works though:

model_tfimm = tfimm.create_model('convnext_base_384_in22ft1k', nb_classes=0, pretrained="timm")

model = tf.keras.models.Sequential()
for layer in model_tfimm.layers:
    model.add(layer)

model.add(tf.keras.layers.Input(shape=(NUM_ADD_CLASSES[0], ), name='input_features_layer'))

model.add(tf.keras.layers.Concatenate([model.layers[-2], model.layers[-1]]))

model.add(tf.keras.layers.Dense(NUM_CLASSES, activation=OUTPUT_ACTIVATION, name='prediction_layer'))

Model initializes without any errors, however, when I try to train this model, it gives me this error, as if it does not see a second Input Layer: "ValueError: Layer sequential expects 1 input(s), but it received 2 input tensors.]()"

The code works if I do not add a second Input layer and a Concatenation layer and do not load an additional data. I call this method: 'predictions = model(prediction_data, training=True)' when trying to train the model, where prediction_data is this list: '[<tf.Tensor 'inputs:0' shape=(6, 384, 384, 3) dtype=float32>, <tf.Tensor 'inputs_2:0' shape=(6, 26) dtype=float32>]'. I tried passing 'prediction_data' as a tuple, but it gives the same error.

[ztsv-av]

So, as far as I understand, it is impossible to train Sequential API model with multiple inputs in my situation, so I need Functional API model. The problem is that 'model_tfimm.output' does not work... not sure how to create Functional model in this situation.

[martinsbruveris]

You have two options:

• You can create a model with nb_classes=0, which will result in a model being created without a classification layer and you can use the usual model output.
• You can call the model with return_features=True and use the key features (should be present for all models)

  model = create_model("resnet18")
  x, features = model(input, return_features=True)
  # features["features"] contains the pre-classifier features.

[ztsv-av]

@martinsbruveris The problem is that though I will not be able to use layers from create_model and train them. I can use the output of this model, but how can I create a new model that uses all layers from create_model and trains them? Additionally, I need to use last layer of create_model to concatenate it with new input for additional features.

[martinsbruveris]

Yes, you should be able to train all layers by default. Consider this code

import tensorflow as tf
import tfimm
from tfimm.utils.flops import get_parameters

x = tf.keras.Input((32, 32, 3))
y = tf.keras.Input((512,))
backbone = tfimm.create_model("resnet18", nb_classes=0)
z = backbone(x)
z = tf.keras.layers.Concatenate()([y, z])
z = tf.keras.layers.Dense(units=10)(z)
model = tf.keras.Model(inputs=[x, y], outputs=z)

print(get_parameters(backbone))
print(get_parameters(model))

Note that the combined model has all the trainable parameters of backbone plus the ones from the dense layer.

[ztsv-av]

@martinsbruveris thank you for your reply. I will try to do that and write you up with results!

[martinsbruveris]

Happy to help. I will close the issue for now. Feel free to reopen it, if needed.