ResNetV2 backbones are incompatible with upsample_type = "upsampling2d"

[ngreenwald]

Describe the bug While trying out some of the other backbones available, I'm running into issues with ResNext and ResNetV2 backbones. The ResNext backbones are loaded without any errors, but using the ImageNet weights fails.

The ResNetV2 backbones can't be loaded at all.

To Reproduce Attempting to use ResNetV2 like this

from deepcell.model_zoo.panopticnet import PanopticNet

new_model = PanopticNet(
    backbone='resnet50v2',
    input_shape=(256, 256, 2),
    norm_method=None,
    num_semantic_heads=2,
    num_semantic_classes=[1, 3], # inner distance, outer distance, fgbg, pixelwise
    location=True,  # should always be true
    include_top=True,
    use_imagenet=False)

Produces the following:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-25-9c416ed80854> in <module>
      9     location=True,  # should always be true
     10     include_top=True,
---> 11     use_imagenet=False)

/usr/local/lib/python3.6/dist-packages/deepcell/model_zoo/panopticnet.py in PanopticNet(backbone, input_shape, inputs, backbone_levels, pyramid_levels, create_pyramid_features, create_semantic_head, frames_per_batch, temporal_mode, num_semantic_heads, num_semantic_classes, required_channels, norm_method, pooling, location, use_imagenet, lite, upsample_type, interpolation, name, z_axis_convolutions, **kwargs)
    285                                            interpolation=interpolation,
    286                                            upsample_type=upsample_type,
--> 287                                            z_axis_convolutions=z_axis_convolutions)
    288 
    289     features = [pyramid_dict[key] for key in pyramid_levels]

/usr/local/lib/python3.6/dist-packages/deepcell/model_zoo/fpn.py in __create_pyramid_features(backbone_dict, ndim, feature_size, include_final_layers, lite, upsample_type, interpolation, z_axis_convolutions)
    264                                       lite=lite,
    265                                       interpolation=interpolation,
--> 266                                       z_axis_convolutions=z_axis_convolutions)
    267         pyramid_finals.append(pf)
    268         pyramid_upsamples.append(pu)

/usr/local/lib/python3.6/dist-packages/deepcell/model_zoo/fpn.py in create_pyramid_level(backbone_input, upsamplelike_input, addition_input, upsample_type, level, ndim, lite, interpolation, feature_size, z_axis_convolutions)
    123     # Add and then 3x3 conv
    124     if addition_input is not None:
--> 125         pyramid = Add(name=addition_name)([pyramid, addition_input])
    126 
    127     # Upsample pyramid input

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    589           # Build layer if applicable (if the `build` method has been
    590           # overridden).
--> 591           self._maybe_build(inputs)
    592 
    593           # Wrapping `call` function in autograph to allow for dynamic control

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in _maybe_build(self, inputs)
   1879       # operations.
   1880       with tf_utils.maybe_init_scope(self):
-> 1881         self.build(input_shapes)
   1882     # We must set self.built since user defined build functions are not
   1883     # constrained to set self.built.

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/utils/tf_utils.py in wrapper(instance, input_shape)
    293     if input_shape is not None:
    294       input_shape = convert_shapes(input_shape, to_tuples=True)
--> 295     output_shape = fn(instance, input_shape)
    296     # Return shapes from `fn` as TensorShapes.
    297     if output_shape is not None:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/merge.py in build(self, input_shape)
    109       else:
    110         shape = input_shape[i][1:]
--> 111       output_shape = self._compute_elemwise_op_output_shape(output_shape, shape)
    112     # If the inputs have different ranks, we have to reshape them
    113     # to make them broadcastable.

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/merge.py in _compute_elemwise_op_output_shape(self, shape1, shape2)
     80           raise ValueError(
     81               'Operands could not be broadcast '
---> 82               'together with shapes ' + str(shape1) + ' ' + str(shape2))
     83         output_shape.append(i)
     84     return tuple(output_shape)

ValueError: Operands could not be broadcast together with shapes (8, 8, 256) (16, 16, 256)

Attempting to use ResNext like this:

from deepcell.model_zoo.panopticnet import PanopticNet

new_model = PanopticNet(
    backbone='resnext50',
    input_shape=(256, 256, 2),
    norm_method=None,
    num_semantic_heads=2,
    num_semantic_classes=[1, 3], # inner distance, outer distance, fgbg, pixelwise
    location=True,  # should always be true
    include_top=True,
    use_imagenet=True)

Produces the following:

ValueError                                Traceback (most recent call last)
<ipython-input-27-18bc2bcf6d73> in <module>
      9     location=True,  # should always be true
     10     include_top=True,
---> 11     use_imagenet=True)

/usr/local/lib/python3.6/dist-packages/deepcell/model_zoo/panopticnet.py in PanopticNet(backbone, input_shape, inputs, backbone_levels, pyramid_levels, create_pyramid_features, create_semantic_head, frames_per_batch, temporal_mode, num_semantic_heads, num_semantic_classes, required_channels, norm_method, pooling, location, use_imagenet, lite, upsample_type, interpolation, name, z_axis_convolutions, **kwargs)
    273                                     frames_per_batch=frames_per_batch,
    274                                     return_dict=True,
--> 275                                     **model_kwargs)
    276 
    277     backbone_dict_reduced = {k: backbone_dict[k] for k in backbone_dict

/usr/local/lib/python3.6/dist-packages/deepcell/utils/backbone_utils.py in get_backbone(backbone, input_tensor, input_shape, use_imagenet, return_dict, frames_per_batch, **kwargs)
    386         # Set the weights of the model if requested
    387         if use_imagenet:
--> 388             model_with_weights = model_cls(**kwargs_with_weights)
    389             model_with_weights.save_weights('model_weights.h5')
    390             model.load_weights('model_weights.h5', by_name=True)

/usr/local/lib/python3.6/dist-packages/keras_applications/resnet_common.py in ResNeXt50(include_top, weights, input_tensor, input_shape, pooling, classes, **kwargs)
    553                   input_tensor, input_shape,
    554                   pooling, classes,
--> 555                   **kwargs)
    556 
    557 

/usr/local/lib/python3.6/dist-packages/keras_applications/resnet_common.py in ResNet(stack_fn, preact, use_bias, model_name, include_top, weights, input_tensor, input_shape, pooling, classes, **kwargs)
    371     x = layers.MaxPooling2D(3, strides=2, name='pool1_pool')(x)
    372 
--> 373     x = stack_fn(x)
    374 
    375     if preact is True:

/usr/local/lib/python3.6/dist-packages/keras_applications/resnet_common.py in stack_fn(x)
    544               **kwargs):
    545     def stack_fn(x):
--> 546         x = stack3(x, 128, 3, stride1=1, name='conv2')
    547         x = stack3(x, 256, 4, name='conv3')
    548         x = stack3(x, 512, 6, name='conv4')

/usr/local/lib/python3.6/dist-packages/keras_applications/resnet_common.py in stack3(x, filters, blocks, stride1, groups, name)
    262         Output tensor for the stacked blocks.
    263     """
--> 264     x = block3(x, filters, stride=stride1, groups=groups, name=name + '_block1')
    265     for i in range(2, blocks + 1):
    266         x = block3(x, filters, groups=groups, conv_shortcut=False,

/usr/local/lib/python3.6/dist-packages/keras_applications/resnet_common.py in block3(x, filters, kernel_size, stride, groups, conv_shortcut, name)
    229                                use_bias=False, name=name + '_2_conv')(x)
    230     x_shape = backend.int_shape(x)[1:-1]
--> 231     x = layers.Reshape(x_shape + (groups, c, c))(x)
    232     output_shape = x_shape + (groups, c) if backend.backend() == 'theano' else None
    233     x = layers.Lambda(lambda x: sum([x[:, :, :, :, i] for i in range(c)]),

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    632                     outputs = base_layer_utils.mark_as_return(outputs, acd)
    633                 else:
--> 634                   outputs = call_fn(inputs, *args, **kwargs)
    635 
    636             except TypeError as e:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/core.py in call(self, inputs)
    465   def call(self, inputs):
    466     return array_ops.reshape(inputs,
--> 467                              (array_ops.shape(inputs)[0],) + self.target_shape)
    468 
    469   def get_config(self):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_array_ops.py in reshape(tensor, shape, name)
   7713   try:
   7714     _, _, _op = _op_def_lib._apply_op_helper(
-> 7715         "Reshape", tensor=tensor, shape=shape, name=name)
   7716   except (TypeError, ValueError):
   7717     result = _dispatch.dispatch(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)
    543               raise ValueError(
    544                   "Tried to convert '%s' to a tensor and failed. Error: %s" %
--> 545                   (input_name, err))
    546             prefix = ("Input '%s' of '%s' Op has type %s that does not match" %
    547                       (input_name, op_type_name, observed))

ValueError: Tried to convert 'shape' to a tensor and failed. Error: None values not supported.

[willgraf]

I can reproduce the errors on both ResNetV2 and ResNext.

[willgraf]

I am not sure why we are seeing the problem with ResNext, it's failing to instantiate the model using its own pre-trained weights. Looking into TF2.3 to see if there were any future bugfixes, I notice that ResNext is not supported.

[willgraf]

ResNetV2 works as expected when using upsample_type = 'upsamplelike'.

When we create_pyramid_level creates pyramid upsamples that are an unexpected shape when using upsampling2d, specifically we are adding the following layers:

Tensor("conv4_block23_out/add:0", shape=(?, 4, 4, 1024), dtype=float32) Tensor("P5_upsampled_1/ResizeBilinear:0", shape=(?, 8, 8, 256), dtype=float32)

However when using "upsamplelike", the upsampled tensor looks like: Tensor("P5_upsampled_1/resize/ResizeNearestNeighbor:0", shape=(?, ?, ?, 256), dtype=float32)

UpsampleLike keeps the x and y dimensions undefined, which allows us to add these features together.