webcam demo results in black segmentation images

[jboys]

I have followed the instructions given by the SegNet Tutorial along with the README.md to run Scripts/webcam_demo.py on a KITTI image sequence.

I made the necessary changes to Scripts/webcam_demo.py to:

• point to my caffe-segnet
• read in the image sequence files rather than webcam frames
• save the segmentation results to disk rather than displayed in a cv2.namedWindow

I also downloaded the weights and executed the script exactly as given in the readme file.

The script iterates over all images in the sequence and saves them to disk, however all segmentation results are just black images.

As a simple diagnostic, I saved the resized input image to disk, which appear normal.

Opening the black segmentation image within Matlab and inspecting the pixel channel values reveals that the values consist of zeros and ones without any discernible pattern.

[arushk1]

Can you post your modified webcam_demo.py file here?

[jboys]

Here is the modified webcam_demo.py script:

import numpy as np
import matplotlib.pyplot as plt
import os.path
import scipy
import argparse
import math
import cv2
import sys
import time

sys.path.append('/home/josh/software/anaconda/lib/python2.7/site-packages') #('/usr/local/lib/python2.7/site-packages')
# Make sure that caffe is on the python path:
caffe_root = '/home/josh/code/segnet/caffe-segnet/'
sys.path.insert(0, caffe_root + 'python')
import caffe

# Import arguments
parser = argparse.ArgumentParser()
parser.add_argument('--model', type=str, required=True)
parser.add_argument('--weights', type=str, required=True)
parser.add_argument('--colours', type=str, required=True)
args = parser.parse_args()

net = caffe.Net(args.model,
                args.weights,
                caffe.TEST)

caffe.set_mode_gpu()

input_shape = net.blobs['data'].data.shape
output_shape = net.blobs['argmax'].data.shape

label_colours = cv2.imread(args.colours).astype(np.uint8)

#cv2.namedWindow("Input")
#cv2.namedWindow("SegNet")

cap = cv2.VideoCapture('/home/josh/dataset_josh/images/test/01/%010d.png')
# Change this to your webcam ID, or file name for your video file

if cap.isOpened(): # try to get the first frame
    rval, frame = cap.read()
else:
    rval = False

count = 0
while rval:
    start = time.time()
    rval, frame = cap.read()
    end = time.time()
    print '%30s' % 'Grabbed camera frame in ', str((end - start)*1000), 'ms'

    start = time.time()
    frame = cv2.resize(frame, (input_shape[3],input_shape[2]))
    input_image = frame.transpose((2,0,1))
    input_image = input_image[(2,1,0),:,:]
    input_image = np.asarray([input_image])
    end = time.time()
    print '%30s' % 'Resized image in ', str((end - start)*1000), 'ms'

    start = time.time()
    out = net.forward_all(data=input_image)
    end = time.time()
    print '%30s' % 'Executed SegNet in ', str((end - start)*1000), 'ms'

    start = time.time()
    segmentation_ind = np.squeeze(net.blobs['argmax'].data)
    segmentation_ind_3ch = np.resize(segmentation_ind,(3,input_shape[2],input_shape[3]))
    segmentation_ind_3ch = segmentation_ind_3ch.transpose(1,2,0).astype(np.uint8)
    segmentation_rgb = np.zeros(segmentation_ind_3ch.shape, dtype=np.uint8)

    cv2.LUT(segmentation_ind_3ch,label_colours,segmentation_rgb)
    segmentation_rgb = segmentation_rgb.astype(float)/255

    end = time.time()
    print '%30s' % 'Processed results in ', str((end - start)*1000), 'ms\n'

    #cv2.imshow("Input", frame)
    #cv2.imshow("SegNet", segmentation_rgb)

    rgbfile = '/home/josh/code/segnet/output/frame_%03d.png' % count
    cv2.imwrite(rgbfile, frame)

    segfile = '/home/josh/code/segnet/output/segnet_%03d.png' % count
    cv2.imwrite(segfile, segmentation_rgb)

    count = count + 1

    #key = cv2.waitKey(1)
    #if key == 27: # exit on ESC
        #break
cap.release()
#cv2.destroyAllWindows()

[arushk1]

I think there's a problem with your videocapture object.

[jboys]

I am able to read in the input image sequence using the VideoCapture object without issue. The frame that is saved to rgbfile looks normal for all input images -- its just a resized version of the original input image (as expected).

[ghost]

With what arguments do you call the webcam_demo.py?

[ghost]

The images are not purely black. You can open them with gimp and apply the levels filter, whcih restricts the range to 0..1 (or so). Then you will see the coloured segments. Or just uncomment this line:

segmentation_rgb = segmentation_rgb.astype(float)/255