Always getting image with zero MAP after inference

[ngoyal2707]

I am using the library to post process road segmentation, which is just pixel-level binary classification problem, to classify each pixel as 'road' or 'no road'.

My code looks as following:

pil_im = np.asarray(Image.open('example_image.jpg', 'r'))
probs = model.predict(pil_im.reshape((1,1024,1024,3))/255)

unary = softmax_to_unary(probs)
unary = np.ascontiguousarray(unary)

d = dcrf.DenseCRF(pil_im.shape[0] * pil_im.shape[1], 1)
d.setUnaryEnergy(unary)

feats = create_pairwise_gaussian(sdims=(10, 10), shape=pil_im.shape[:2])

d.addPairwiseEnergy(feats, compat=3,
                    kernel=dcrf.DIAG_KERNEL,
                    normalization=dcrf.NORMALIZE_SYMMETRIC)
feats = create_pairwise_bilateral(sdims=(50, 50), schan=(20, 20, 20),
                                   img=pil_im, chdim=2)
d.addPairwiseEnergy(feats, compat=10,
                     kernel=dcrf.DIAG_KERNEL,
                     normalization=dcrf.NORMALIZE_SYMMETRIC)

Q = d.inference(10)
res = np.argmax(Q, axis=0).reshape((pil_im.shape[0], pil_im.shape[1]))
print(res.max())

I always get res to be matrix of 0.0 . I have tried following things: 1) Change it to use 2 channels with probability of foreground and background. 2) Changing sdims params. 3) Running inference longer.

Nothing helps and I always get matrix with 0.0 values.

If it helps, pil_img looks like this and probs looks like this after thresholding at 0.5.

Also I tried to print KL values at each step of inference. Doesn't seem like it's improving at all.

KL-divergence at 0: 30498.999462480373
KL-divergence at 1: 30498.999462480373
KL-divergence at 2: 30498.999462480373
KL-divergence at 3: 30498.999462480373
KL-divergence at 4: 30498.999462480373
KL-divergence at 5: 30498.999462480373
KL-divergence at 6: 30498.999462480373
KL-divergence at 7: 30498.999462480373
KL-divergence at 8: 30498.999462480373
KL-divergence at 9: 30498.999462480373
KL-divergence at 10: 30498.999462480373
KL-divergence at 11: 30498.999462480373
KL-divergence at 12: 30498.999462480373
KL-divergence at 13: 30498.999462480373
KL-divergence at 14: 30498.999462480373
KL-divergence at 15: 30498.999462480373
KL-divergence at 16: 30498.999462480373
KL-divergence at 17: 30498.999462480373
KL-divergence at 18: 30498.999462480373
KL-divergence at 19: 30498.999462480373

EDIT: I tried with original cpp code by changing the label counts M to be 2 and making the library. I tried both dense_learning.cpp and dense_inference.cpp using following command:

./dense_learning image.ppm mask.ppm final_mask.ppm

I still get the final image as just background color.

[lucasb-eyer]

Thanks, I'm still in the middle of a cross-country move, so it might take a little until I can have a look, but the full code and images will help a lot in helping you here :smile:

[ximua00]

I am having the same issue… any updates?

[lucasb-eyer]

Hi, I'm very late here, but there we go.

Your data is very problematic, I'm attaching a screenshot of what your images actually look like at the end of this answer. Basically, they include an alpha channel, which hides the fact that you are saving the axis, axis-ticks and axis-labels along with the images, including in the probabilities probs!

This indicates to me that there's a serious mistake somewhere in your pipeline.

Next, I don't understand why you're not using the dcrf.DenseCRF2D API, since you are dealing with 2D images. But nevertheless, the generic N-D API that you are using will still work.

You mentioned that you ran it as a 2-class problem (which is what I would recommend) and it doesn't work. Well, with your broken images (see below), it "works" as in it produces two classes as a result, although you can see that it does indeed get rid of the road.

After that, I think it is just a matter of hyperparameter search, as looking at your image, it could be very difficult to find a case where denseCRF actually helps, since it usually "snaps" labels to colors. See the last screenshot below for an example of such a hyperparameter sweep, I immediately find a "working" example, though you might want to test for quality.

[Work-jk-l]

@lucasb-eyer My Project is based deeplab v3+ to classify 2 labels，after the process in deeplab v3+,the edges in the prediction image looks bad. And I need the accurate edges . After i run the inference.py in expamples. The final result color is not just 2 labels.because the final images color ivaries between [1,1,1] and [foreground color], I am curious that shout it just be 2 colors in the final images:One is background ,One is the foreground. Ths a lot!