How to implement the improvement of build_fpn_mask_graph?

[John1231983]

Hello all, I have a plane to implement an improvement of mask segmentation _branch. It is described as following:

We further create a short path from layer conv3 to a fc layer. There are two 3×3 convolutional layers where the second shrinks channels to half to reduce computational overhead. The mask size we use is 28 × 28 so that the fc layer produces a 784 × 1 × 1 vector. This vector is reshaped to the same spatial size as the mask predicted by FCN. To obtain the final mask prediction, mask of each class from FCN and foreground/background prediction from fc are added.

The below code shows my implementation that I want to ask two questions:

1. What is the number of feature of mrcnn_mask_conv4_fc and mrcnn_mask_conv5_fc. The paper said that "two 3×3 convolutional layers where the second shrinks channels". Is it 128?
2. What is the reshape size (after fully connected layer). How can we fill the ? in the x_fc = K.reshape(x_fc, (-1, ?, ?, ?, ?)))?

3. I have run the code but it shows some error likes below. I guess have some problem with Add function

   File "/home/john/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
   return func(*args, **kwargs)
   File "/home/john/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 1732, in __init__
   build_map_of_graph(x, finished_nodes, nodes_in_progress)
   File "/home/john/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 1722, in build_map_of_graph
   layer, node_index, tensor_index)
   File "/home/john/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 1722, in build_map_of_graph
   layer, node_index, tensor_index)
   File "/home/john/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 1693, in build_map_of_graph
   layer, node_index, tensor_index = tensor._keras_history
   AttributeError: 'Tensor' object has no attribute '_keras_history'

   This is my implementation

   x = KL.TimeDistributed(BatchNorm(axis=3),
                          name='mrcnn_mask_bn3')(x)
   x = KL.Activation('relu')(x)
   x_classify = x
   ...
   # Add fc _branch
   x_fc = KL.TimeDistributed(KL.Conv2D(256, (3, 3), padding="same"),
                                 name="mrcnn_mask_conv4_fc")(x_conv3)
   x_fc = KL.Activation('relu')(x_fc)
   x_fc = KL.TimeDistributed(KL.Conv2D(128, (3, 3), padding="same"),
                                 name="mrcnn_mask_conv5_fc")(x_fc)
   x_fc= KL.TimeDistributed(KL.Dense(num_classes, activation='linear'),
                                 name='mrcnn_bbox_fc')(x_fc)
   
   x_fc = K.reshape(x_fc, (-1, ?, ?, ?, ?)))
   x_fc = KL.TimeDistributed(KL.Add(name="mrcnn_bbox_fc_deconv_add")([x_fc, x_classify]))
   x_fc = KL.TimeDistributed(KL.Conv2D(num_classes, (1, 1), strides=1, activation="sigmoid"),
                          name="mrcnn_mask")(x_fc)
    return x_fc

[kongjibai]

Hi John，did you have implemented the plan above? And which paper this plan come from?

[ahuxjz]

大家好，我有一个飞机来实现遮罩分割_branch的改进。描述如下：

我们进一步创建了一条从conv3层到fc层的短路径。有两个3×3卷积层，其中第二个将通道缩小到一半以减少计算开销。我们使用的遮罩大小为 28×28，因此fc层会产生784×1×1的向量。将此矢量重塑为与FCN预测的蒙版相同的空间大小。为了获得最终的掩模预测，添加了来自FCN的每个类别的掩模和来自fc的前景/背景预测。

下面的代码显示了我要问两个问题的实现：

1. mrcnn_mask_conv4_fc和的特征数是多少mrcnn_mask_conv5_fc。论文说“第二个收缩通道的两个3×3卷积层”。是128吗？
2. 整形尺寸是多少（完全连接的层之后）。我们怎样才能填补?的x_fc = K.reshape(x_fc, (-1, ?, ?, ?, ?)))？
3. 我已经运行了代码，但是下面显示了一些错误。我想Add功能有问题

  File "/home/john/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/john/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 1732, in __init__
    build_map_of_graph(x, finished_nodes, nodes_in_progress)
  File "/home/john/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 1722, in build_map_of_graph
    layer, node_index, tensor_index)
  File "/home/john/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 1722, in build_map_of_graph
    layer, node_index, tensor_index)
  File "/home/john/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 1693, in build_map_of_graph
    layer, node_index, tensor_index = tensor._keras_history
AttributeError: 'Tensor' object has no attribute '_keras_history'

这是我的实现

    x = KL.TimeDistributed(BatchNorm(axis=3),
                           name='mrcnn_mask_bn3')(x)
    x = KL.Activation('relu')(x)
    x_classify = x
    ...
    # Add fc _branch
    x_fc = KL.TimeDistributed(KL.Conv2D(256, (3, 3), padding="same"),
                                  name="mrcnn_mask_conv4_fc")(x_conv3)
    x_fc = KL.Activation('relu')(x_fc)
    x_fc = KL.TimeDistributed(KL.Conv2D(128, (3, 3), padding="same"),
                                  name="mrcnn_mask_conv5_fc")(x_fc)
    x_fc= KL.TimeDistributed(KL.Dense(num_classes, activation='linear'),
                                  name='mrcnn_bbox_fc')(x_fc)

    x_fc = K.reshape(x_fc, (-1, ?, ?, ?, ?)))
    x_fc = KL.TimeDistributed(KL.Add(name="mrcnn_bbox_fc_deconv_add")([x_fc, x_classify]))
    x_fc = KL.TimeDistributed(KL.Conv2D(num_classes, (1, 1), strides=1, activation="sigmoid"),
                           name="mrcnn_mask")(x_fc)
     return x_fc

Has your problem been solved? I also encountered this problem when trying to change the network structure.