deeplab doesn't predict correctly the segmentation masks

[kirk86]

Everything seems to be working properly training/evaluation etc. except from the fact that deeplab doesn't predict the segmentation masks.

Example:

The original images in the dataset are either colored like the above one or black and white, but all the masks are black and white.

[IamShubhamGupto]

@kirk86 do you have imbalanced data distribution like 1:100? i encountered your problem before, after setting different weight_loss, i got non-black masks.

Could you show how to set the different weight_loss ? thanks

[dronefreak]

@IamShubhamGupto Here is how you can do it. Under the utils folder, there is a file called train_utils.py There you need to add the following lines after the line scaled_labels = tf.reshape(scaled_labels, shape=[-1])

ignore_weight = 0
label0_weight = 3  # class 1
label1_weight = 5 # class 2
not_ignore_mask = tf.to_float(tf.equal(scaled_labels, 0)) * label0_weight + tf.to_float(tf.equal(scaled_labels, 1)) * label1_weight + tf.to_float(tf.equal(scaled_labels, ignore_label)) * ignore_weight

The weights are experimental, you might need to tweak around to get the right combinations.

[a448262375]

I solved the black image problem, refer to this document: https://blog.csdn.net/u011974639/article/details/80948990

1. Number of class. In my case, I want to detect circle and triangle in images, so I have 2 classes. This case, I should set num_classes = 4, it is 2 classes + background + ignore_label. refer to segmentation_dataset.py , data_generator.py

_ORIGINAL_INFORMATION = DatasetDescriptor( splits_to_sizes={ 'train': 560, #number of images for training 'val': 240, #number of images for test }, num_classes=4, # circle + triangle + background + ignore_label ignore_label=255, # ignore_label means to outline of object. 255 means the white color. ignore it )

2. Training weight Modify the train_utils.py

   irgore_weight = 0 label0_weight = 1 label1_weight = 10 label2_weight = 15

   not_ignore_mask = tf.to_float(tf.equal(scaled_labels, 0)) label0_weight + tf.to_float(tf.equal(scaled_labels, 1)) label1_weight + tf.to_float(tf.equal(scaled_labels, 2)) label2_weight + tf.to_float(tf.equal(scaled_labels, ignore_label)) irgore_weight

   one_hot_labels = tf.one_hot( scaled_labels, num_classes, on_value=1.0, off_value=0.0)

   if top_k_percent_pixels == 1.0: tf.losses.softmax_cross_entropy( one_hot_labels, tf.reshape(logits, shape=[-1, num_classes]), weights=not_ignore_mask, scope=loss_scope)

Additional information: In train.py set: initialize_last_layer=False last_layers_contain_logits_only=True Images and annotation are both .png Annotation images are gray_scale: Class1 = 1, Class2 = 2, BackGround = 0, outline = 1 or 2 (you can set 255, but this time, I set color of outline same to the object) *Total lose is more than 4.0

*Command: python train.py --logtostderr --train_split="train" --model_variant="xception_65" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --train_crop_size=81,81 --train_batch_size=8 --training_number_of_steps=500 --fine_tune_batch_norm=false --tf_initial_checkpoint='./deeplabv3_pascal_train_aug/model.ckpt' --train_logdir="./datasets/datagen/exp/train_on_trainval_set/train" --dataset_dir="./datasets/datagen/tfrecord" --dataset="original"

python vis.py --logtostderr --vis_split="val" --model_variant="xception_65" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --vis_crop_size=81,81 --checkpoint_dir="./datasets/datagen/exp/train_on_trainval_set/train" --vis_logdir="./datasets/datagen/exp/train_on_trainval_set/vis" --dataset_dir="./datasets/datagen/tfrecord" --max_number_of_iterations=1 --dataset=original

[harshgrovr]

Just a small addition.

@a448262375

segmentation_dataset.py has been deprecated we have to make the changes regarding number of classes or adding custom data info in data_generator.py

` _ADE20K_INFORMATION = DatasetDescriptor( splits_to_sizes={ 'train': 20210, # num of samples in images/training 'val': 2000, # num of samples in images/validation }, num_classes=151, ignore_label=0, )

_imageseg_INFORMATION = DatasetDescriptor( splits_to_sizes={ 'train': 34, # num of samples in images/training 'val': 8, # num of samples in images/validation }, num_classes=2, # background with 0 and annotation with 1 ignore_label=255, )

_DATASETS_INFORMATION = { 'cityscapes': _CITYSCAPES_INFORMATION, 'pascal_voc_seg': _PASCAL_VOC_SEG_INFORMATION, 'ade20k': _ADE20K_INFORMATION, 'imageseg':_imageseg_INFORMATION, } `

Follow this link for more info: https://blog.csdn.net/u011974639/article/details/80948990

[dronefreak]

@harshgrovr I think there is a further change in your code. You would need to add the ignore_label as well, in your imageseg dataset.

Otherwise, you might get an error like this: TypeError: __new__() missing 1 required positional argument: 'ignore_label'

The modified code would be:

_imageseg_INFORMATION = DatasetDescriptor(
splits_to_sizes={
'train': 34, # num of samples in images/training
'val': 8, # num of samples in images/validation
},
num_classes=2, # background with 0 and annotation with 1
ignore_label=some_vale, # whatever the label is
)

_DATASETS_INFORMATION = {
'cityscapes': _CITYSCAPES_INFORMATION,
'pascal_voc_seg': _PASCAL_VOC_SEG_INFORMATION,
'ade20k': _ADE20K_INFORMATION,
'imageseg':_imageseg_INFORMATION,
}

[harshgrovr]

@dronefreak yeah forgot to mention. thanks!

[sblackstone]

For those who want to try changing their label weights, there is no longer a need to modify files to do so, you can add options to your command line params:

    --label_weights=1 \
    --label_weights=2 \
    --label_weights=3 \
    --label_weights=4 \
    --label_weights=5 \

Where the n-th --label-weight corresponds to your nth class.

[LightingX]

For those who want to try changing their label weights, there is no longer a need to modify files to do so, you can add options to your command line params:

    --label_weights=1 \
    --label_weights=2 \
    --label_weights=3 \
    --label_weights=4 \
    --label_weights=5 \

Where the n-th --label-weight corresponds to your nth class.

If there are 2 classes in my datasets, and the ratio of bakground (class 0) and foreground (class 1) is 100:1 like this (when trainning, I have transformed the white color to the pixel value 1), does it mean that I should set the lable weihgts with [1, 100]?

[northeastsquare]

I also met this problem. I have two class:background, lane. number of lane's pixel is less, maybe 0.01. I add this in train_utils.py. Note: the deeplab code pad image with 255 ignore label.

  nforeground = tf.reduce_sum(tf.to_float(tf.to_float(labels)>0.1))
  nlane = tf.reduce_sum(tf.to_float(tf.equal(labels, 1)))
  n127 = tf.reduce_sum(tf.to_float(labels>=2))
  n255 = tf.reduce_sum(tf.to_float(tf.equal(labels, 255)))
  h,w = preprocess_utils.resolve_shape(labels, 4)[1:3]
  top_k_percent_pixels = nlane/tf.to_float(h*w)
  top_k_percent_pixels = tf.minimum(tf.maximum(top_k_percent_pixels, 0.00001), 0.7)
  top_k_percent_pixels = tf.Print(top_k_percent_pixels, \
    [top_k_percent_pixels, h,w,nforeground,nlane,n127,n255], "top_k_percent_pixels:", first_n=10)

[aleeminati]

Hello! I have been trying to implement the updated version of deeply with only 1 foreground class and 1 background. Initially, my background was black and the class was white but that wasnot wise so I wrote a script that converted all (255,255,255,255) pixels to (1,1,1,255). However, I am still getting black outputs. My --num-classes=2 and I changed it to -num-classes=3 but no effect. I also put a 1:20 label weight but still no effect.

[Vaishali-Nimilan]

Hello, I have been training my dataset in deeplab, I haven't figured a way to check if the model is overfitting. Is there a way to see the validation graph in tensorboard?

[trongan93]

Hello! I have been trying to implement the updated version of deeply with only 1 foreground class and 1 background. Initially, my background was black and the class was white but that wasnot wise so I wrote a script that converted all (255,255,255,255) pixels to (1,1,1,255). However, I am still getting black outputs. My --num-classes=2 and I changed it to -num-classes=3 but no effect. I also put a 1:20 label weight but still no effect.

I got the same problem, Sol 1: I adjust the num-classes=2 (1 for the foreground object, 0 for background) and num-classes=3 (1 for foreground object, 0 for background, 255 for ignore_label). Both methods are don't make the iou changed. Sol 2: I change the weight_label, 1:10 , 1:100, 0.1:10, 10:1, 100:1, .... All cases not work.

[trongan93]

I also met this problem. I have two class:background, lane. number of lane's pixel is less, maybe 0.01. I add this in train_utils.py. Note: the deeplab code pad image with 255 ignore label.

  nforeground = tf.reduce_sum(tf.to_float(tf.to_float(labels)>0.1))
  nlane = tf.reduce_sum(tf.to_float(tf.equal(labels, 1)))
  n127 = tf.reduce_sum(tf.to_float(labels>=2))
  n255 = tf.reduce_sum(tf.to_float(tf.equal(labels, 255)))
  h,w = preprocess_utils.resolve_shape(labels, 4)[1:3]
  top_k_percent_pixels = nlane/tf.to_float(h*w)
  top_k_percent_pixels = tf.minimum(tf.maximum(top_k_percent_pixels, 0.00001), 0.7)
  top_k_percent_pixels = tf.Print(top_k_percent_pixels, \
    [top_k_percent_pixels, h,w,nforeground,nlane,n127,n255], "top_k_percent_pixels:", first_n=10)

Hi, I'm missing the same with your problem, segmented objects ratio (foreground : background) is a big difference. Could you please explain the detailed where to add your code to the train_utils.py file? Thanks!

[sunzhe09]

I met the same problem

[gouri1694]

Hi, There is no prediction code available with deeplabv3+ semantic segmentation. Can somebody please share if it's available or any reference, I have trained the model but I want prediction code, I don't have much knowledge on tensorflow

[edsumpena]

@gouri1694 Use vis.py to visualize the output like the following: !python /content/models/research/deeplab/vis.py --logtostderr \ --vis_split="val" \ --model_variant="mobilenet_v2" \ --vis_crop_size="513,513" \ --min_resize_value=513 \ --max_resize_value=513 \ --dataset="pascal_voc_seg" \ --checkpoint_dir="/checkpoint" \ --vis_logdir="/outputs" \ --dataset_dir="/tfrecord" \ --max_number_of_iterations=1 --eval_interval_secs=0 This will generate the input images and output masks in the "/outputs" folder.