Running a dense CRF on classifier predictions from a neural network

[ghost]

Hi,

Thank you for sharing this code. I have a deep learning classifier that specifies probability values for three classes at each pixel in an image. I would like to run the dense CRF to clean up my predictions. Here is the code that I'm trying to use.

`

Define the CRF (width, height, nlabels)

Predictions is a numpy array from my classifier, for a three class classification problem

d = dcrf.DenseCRF2D(predictions.shape[1], predictions.shape[0], 3)

Unary potentials

U = predictions.reshape((3, -1)) d.setUnaryEnergy(-np.log(U))

Pairwise potentials

d.addPairwiseGaussian(sxy=(3,3), compat=3, kernel=dcrf.DIAG_KERNEL, normalization = dcrf.NORMALIZE_SYMMETRIC)

Inference

Q = d.inference(5)

map = np.argmax(Q, axis=0).reshape((predictions.shape[0], predictions.shape[1]))

proba = np.array(map)

Write to a MAT file

scipy.io.savemat("CRF_Predictions.mat", mdict={'proba': proba})`

Here is my original predictions:

This is the ground truth:

This is what I get after I run the code above:

If anyone can tell me where I am making a mistake, that will be really helpful.

Thanks, Chaitanya

[lucasb-eyer]

My guess is that you are screwing up the reshaping and/or sizes at least once, maybe multiple times. Just briefly looking at your code, I can already see that you're creating a DenseCRF of shape predictions.shape[1], predictions.shape[0] in the constructor, but after mAP reshaping the result to predictions.shape[0], predictions.shape[1]), which doesn't make sense. If you look closely at the README or examples, you should see that the shape is always consisent. Maybe it's a typo in the constructor call?

[ghost]

Thanks for writing back. I assumed that the constructor takes in width and height as the first and second arguments respectively and that for a numpy array the first size is the number of rows (height) and the second size is the number of cols (width). Maybe I got myself confused there.

I fixed the code so that the order of arguments to the constructor and MAP reshaping are predictions.shape[0], predictions.shape[1] now. I still get the same result though.

[lucasb-eyer]

Hmm, if you could share everything I need to reproduce (including the predictions file), I could try to have a look tomorrow, not sure if I'll find the time though.

I'm still pretty sure you're having a mixup of shapes/dimensions/order somewhere. I've seen the same mistake here before, with similar looks.

[ghost]

Thanks, I've attached my code and the predictions file here. I will also check my code again for any errors.

CRF example.zip

[lucasb-eyer]

Ah, yes, the problem lies in this line:

U = predictions.reshape((3, -1))

Because predictions has shape H,W,3, you cannot simply reshape it to 3,-1 and expect that to work! You need to move the last axis to the front first, for example like so:

U = predictions.transpose(2,0,1).reshape((3, -1))

and then it will work, I just tried it. There is no way I can detect that in PyDenseCRF, but I will add it to the FAQ and make it a bit clearer in the README, but really, you are misunderstanding NumPy here.

[lucasb-eyer]

Oh, and your original constructor call of d = dcrf.DenseCRF2D(predictions.shape[1], predictions.shape[0], 3) was correct, it seems.

[ghost]

I made the fixes you’ve mentioned and it works perfectly now. Thank you.