Append new layers into the existing deeplab existing symbol to enahnce result...Is it possible?

[chowkamlee81]

Hai,

I want incorporate upsampling into deeplab semantic segmentation module. Does the same pretrained module work? since iam getting crash @ line 100 in deeplab/train.py file ""sym_instance.init_weights(config, arg_params, aux_params)"" Whether i can add few more layers to enhance this current framework?Is it possible. Do i need to change the pretrained model also? Kindly suggest. Need your inputs

[franciszzj]

1. You don't need to change the pretrained model.
2. You need to initialize your new layer in the symbol file first, or just create a python layer.

[chowkamlee81]

Hai, In deeplab/symbol/resnet_v1_101_deeplab.py, i added few modules for enhancement code below. After adding the code below mentioned, i got exception at sym_instance.init_weights(config, arg_params, aux_params) in the file deeplab/deeplab_train_test.py Exception arg_params['score_weight'] = mx.random.normal(0, 0.01, shape=self.arg_shape_dict['score_weight']) KeyError: 'score_weight'

Pls kindly suggest since i really don't how to proceed. I got messes up. Pls help Code: upscale_factor = 16 upsampling = mx.symbol.Deconvolution(data=score, num_filter=num_classes, kernel=(32, 32), stride=(16, 16), num_group=num_classes, no_bias=True, name='upsampling', attr={'lr_mult': '0.0'},workspace=self.workspace) interim_featmap = mx.symbol.Convolution(name='interim_featmap', data=relu_fc6, num_filter=upscale_factorupscale_factornum_classes, pad=(0,0), kernel=(1, 1), stride=(1, 1),no_bias=True) splitted = mx.symbol.reshape(name='splitted', data=interim_featmap, shape=(0, -4, -1, upscale_factor*2, 0, 0)) unflatten = mx.symbol.reshape(name='unflatten', data=splitted, shape=(0, 0, -4, upscale_factor, upscale_factor, 0, 0)) swapped = mx.symbol.transpose(name='swapped', data=unflatten, axes=(0, 1, 2, 4, 3, 5)) upsampling_featmap = mx.symbol.reshape(name='upsampling_featmap', data=swapped, shape=(0, 0, -3, -3)) label_map = mx.symbol.max(name='maximum',data = upsampling_featmap, axis=1, exclude=False) croped_score = mx.symbol.Crop([upsampling_featmap, data], offset=(8, 8), name='croped_score') softmax = mx.symbol.SoftmaxOutput(data=croped_score, label=seg_cls_gt, normalization='valid', multi_output=True, use_ignore=True, ignore_label=255, name="softmax")

[franciszzj]

Layer name need to match! There is no layer named 'score', right?

[chowkamlee81]

Instead of using deconvolution i want to use Dense upsampling convolution. I retained the previous code but i didn't use the 'score' layer So i used the code below only for the last few layers. Kindly look at the below code, pls suggest and help

Code

1....) interim_featmap = mx.symbol.Convolution(name='interim_featmap', data=relu_fc6, num_filter=upscale_factorupscale_factornum_classes, pad=(0,0), kernel=(1, 1), stride=(1, 1),no_bias=True) 2....) splitted = mx.symbol.reshape(name='splitted', data=interim_featmap, shape=(0, -4, -1, upscale_factor*2, 0, 0)) 3....) unflatten = mx.symbol.reshape(name='unflatten', data=splitted, shape=(0, 0, -4, upscale_factor, upscale_factor, 0, 0)) 4....) swapped = mx.symbol.transpose(name='swapped', data=unflatten, axes=(0, 1, 2, 4, 3, 5)) upsampling_featmap = mx.symbol.reshape(name='upsampling_featmap', data=swapped, shape=(0, 0, -3, -3))label_map = mx.symbol.max(name='maximum',data = upsampling_featmap, axis=1, exclude=False) 5....) croped_score = mx.symbol.Crop([upsampling_featmap, data], offset=(8, 8), name='croped_score') 6.....) softmax = mx.symbol.SoftmaxOutput(data=croped_score, label=seg_cls_gt, normalization='valid', multi_output=True, use_ignore=True, ignore_label=255, name="softmax")

[chowkamlee81]

Kindly pls help

[chowkamlee81]

@Oh233 @StOnEGiggity @Franciszzj @terrychenism @terrychenism @YuwenXiong @ancientmooner Kindly help for the above problem

[StOnEGiggity]

I think this error may occur because you init layer named "score" from old network and in your new symbol, you have removed it. So, just remove anything about score in your init function, I guess.

[chowkamlee81]

Yeah i have removed score_weight,score_bias and upsampling_weight in init function. Now i replaced with my new layer weights below: Only the weights interim_featmap_weight is considered and the rest of the weights are ignored. Am i doing correct?

Kindly pls help me out.....

New code below with weights in init function arg_params['interim_featmap_weight'] = mx.nd.zeros(shape=self.arg_shape_dict['interim_featmap_weight']) arg_params['splitted_weight'] = mx.nd.zeros(shape=self.arg_shape_dict['splitted_weight']) arg_params['unflatten_weight'] = mx.nd.zeros(shape=self.arg_shape_dict['unflatten_weight']) arg_params['swapped_weight'] = mx.nd.zeros(shape=self.arg_shape_dict['swapped_weight']) arg_params['upsampling_featmap_weight'] = mx.nd.zeros(shape=self.arg_shape_dict['upsampling_featmap_weight'])

[StOnEGiggity]

I couldn't understand what you want to do clearly. But if you init weights with zero, it may learn one feature only.

[chowkamlee81]

Yeah let me understand you.. I want to improve deeplab+deform conv results with my own logic. a)In the top few layers i woulld like GCN according to "Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network" in first top few layers rather than RESNET b) One more possibility is to replace deconvolution filter(upsampling stage) by dense upsampling filters according to image-supersolution filter. This is my objective below .... One more thing i observed is the pretrained_model/resnet_v1_101-0000.params, this params has the same symbol deeplab/symbols/resnet_v1_101_deeplab.py. My question is that if use my own symbol, then weights initialized from pretrained_model/resnet_v1_101-0000.params won't be in synchronous with my layers at the top and bottom portion of deeplab symbol. Hence kindly suggest whether we had to use pretrained_model/resnet_v1_101-0000.params if we use our own symbol.

[chowkamlee81]

@StOnEGiggity can you pls help me on this issue

[StOnEGiggity]

You should understand the weights initialized by name, so if you have the same name in your new symbol, the pretrain model will work and check the shape of the same layer should be matched. Maybe you could check the code in MXNet about load params and understand the arguments,like allow_missing. Good luck for you:)