Application to 2d image segmentation

[JulienSiems]

Hi Miguel!

You wrote in the readme that you found that your crf-rnn didn't make any difference when applied to MRI and CRT scans. Have you benchmarked your layer on 2d image segmentation models? Did it work?

Many thanks, Julien

[MiguelMonteiro]

Hi,

I did not benchmark on 2D image segmentation.

I did add noise to the output of the convolutional part of the network (with 3D images) to see if the CRFasRNN layer was able to remove it and maintain the segmentation quality, which it was. I did this to see if the code was able to "clean" the labels using proximity and color, indicating whether or not the implementation was correct.

If you are interested in using this for 2D segmentation I would be glad to give support. Some external validation of the results reported in the original paper would be great. Often enough the performance of the "new" algorithm is over-reported and the "old" under-reported creating miss-leading results.

[JulienSiems]

Hi! Thanks for the quick response. I have been working on applying a crf-rnn layer to my network which works on single instance images (The image contains only one object and some background). Initially, I integrated the crf-rnn-keras from here. The resulting segmentations showed contours and strong edges from the image.

I tried yours today and I couldn't find any sign that it took the reference image into account. Should the images in your case be normalized between 0 and 255 or 0 and 1? From your examples it seems like it doesn't matter. I also tried out your theta_alpha, theta_beta, theta_gamma but that didn't make any difference in my case.

[MiguelMonteiro]

Interesting... It shouldn't really matter if the images are normalized or not provided you scale theta appropriately. Did you try the GPU or CPU version? If you used the GPU version did you compile it for the right number of spatial dimensions? Do you have some example code comparing the two repos I could look at?

[JulienSiems]

I used the GPU version and compiled with SPATIAL_DIMS=2, INPUT_CHANNELS=2 (object/background) and REFERENCE_CHANNELS=3. In crf_rnn_layer_test_2.py you used an rgb image as reference which appears to be within [0, 255] when fed to the crf-rnn layer, so I used theta_alpha = 8, theta_beta = 0.125, theta_gamma = 2. I'll have another training run over the night, to check the parameters again because I changed my code a lot today. The CRF-RNN integration is part of another project of which I can't publish the code. However, I modified your layer to have the same API to the crf-rnn-keras layer, which in turn I modified slightly to be pure TensorFlow.

[MiguelMonteiro]

The values seem right.

Those tests are just examples on how to use it. They aren't supposed to do anything really, since the inputs/outputs have little meaning. It is easier to look at the tests from my permutohedral lattice repo since those actually do filtering.

The way way the code is implemented these two should be the same:

1. Image [0-255]; theta_beta = 1
2. Image [0-1]; theta_beta = 255 The same is not true for theta_alpha and theta_gamma which control the importance of proximity in space. So these must be adjusted depending on the size of the input.

Now that you have mentioned this, I am second guessing my implementation. I was already writing a paper on how this method does not improve segmentation on 3D images... I will build a toy program for 2D to see if it works as expected. Let me know if you would like to help we can talk by email.

[JulienSiems]

Great! My email is juliensiems@gmail.com

[MiguelMonteiro]

I have sent you some emails, it seems the issue is the choice of theta

[netw0rkf10w]

Any updates on this, @JulienSiems and @MiguelMonteiro ? I have tried putting Miguel's implementation on top a CNN but it did not seem to do anything.

What I did is replacing the CRF in sadeepj's repo with Miguel's, and used the same demo code provided by sadeepj.

I tested three models: FCN alone, FCN + sadeepj's CRF, and FCN + Miguel's CRF. I loaded the weights of FCN into all the three models (that means the CRF part in the two last models takes their default weights with theta_alpha=160, theta_beta=3, theta_gamma=3).

Results: FCN + sadeepj's CRF produced segmentation with strong edges, while the segmentations by FCN + Miguel's CRF and FCN were (visually) identical!

[MiguelMonteiro]

Thank you for testing this. I have some questions that might help us get to the problem. 1) Did you train all models from random initialization? Or did you use their pertained model? 2) Did you use the same thetas for both implementations? 3) Can you provide the code? 4) Did you compile the permutohedral lattice for the correct dimensions? 5) Did you test the compiled lattice just for bilateral filtering to see if it is working?

Best,

Miguel

On Tue, 13 Nov 2018, 00:22 D. Khuê Lê-Huu <notifications@github.com wrote:

Any updates on this, @JulienSiems https://github.com/JulienSiems and @MiguelMonteiro https://github.com/MiguelMonteiro ? I have tried putting Miguel's implementation on top a CNN but it did not seem to do anything.

What I did is replacing the CRF in sadeepj's repo https://github.com/sadeepj/crfasrnn_keras/blob/216d88ebe84282ee628df93cf0d452842187be24/src/crfrnn_model.py#L105 with Miguel's https://github.com/MiguelMonteiro/CRFasRNNLayer/blob/a5a1649dd5388946173257f20f897bbfd93d462d/crf_as_rnn_keras_layer.py#L32, and used the same demo code provided by sadeepj.

I tested three models: FCN alone, FCN + sadeepj's CRF, and FCN + Miguel's CRF. I loaded the weights of FCN into all the three models (that means the CRF part in the two last models takes their default weights with theta_alpha=160, theta_beta=3, theta_gamma=3).

Results: FCN + sadeepj's CRF produced segmentation with strong edges, while the segmentations by FCN + Miguel's CRF and FCN were (visually) identical!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MiguelMonteiro/CRFasRNNLayer/issues/1#issuecomment-438060026, or mute the thread https://github.com/notifications/unsubscribe-auth/AcMye0l_v4kLVBvmOpM49dE-cqduYRjwks5uuf2ngaJpZM4Ucel7 .

[netw0rkf10w]

Hi Miguel,

Let me answer your questions: 1) No I did not train anything. What I did is loading the pretrained FCN weights into the models, i.e. only the FCN part has trained parameters, the CRF doesn't. I guess in this case the CRF has its initialized parameters. In your implementation they are initialized with tf.initializers.truncated_normal(mean=0, stddev=0.1) while in sadeepj's, it's identity matrix for the kernel weights, and Potts for the compatibility matrix. I tried setting the initializer in your layer to Potts but this did not change anything. It should be noted that sadeepj's CRF layer also does not have trained parameters but still produced good results. 2) Yes thetas are the same: theta_alpha=160, theta_beta=3, theta_gamma=3. 3) Yes I will prepare and upload it later today. 4) I think the dimensions are correct (color image and 21 classes):

SPATIAL_DIMS=2
INPUT_CHANNELS=21
REFERENCE_CHANNELS=3

I did not encounter any error when running. 5) Yes, using the script Tests/bilateral_filter.py. For this I had to re-compile again with

SPATIAL_DIMS=2
INPUT_CHANNELS=3
REFERENCE_CHANNELS=3

Running Tests/bilateral_filter.py I obtained the same results as shown in your README.

Thanks again!

[netw0rkf10w]

For debugging I tried to print the energy at each iteration, but failed :( I tried this code:

energy = tf.reduce_sum(tf.multiply(tf.subtract(0.5*pairwise, unaries), x)) 
print_energy_keras = keras.backend.print_tensor(energy, message="energy: ")
print_energy_tf = tf.Print(energy, [energy], message="energy: ")

but it did not print anything. Do you have idea on this, Miguel? Thanks.

[MiguelMonteiro]

So here is one big issue: The CRF does have trainable parameters that must be optmised for within the training loop. When you load their model you are loading their CRF weights and you end up with a complete model which does what it is supposed to. When you initialise my version of the CRF with random weights then it is doing nothing but random calculations... I am afraid I don't have any pre-trained weights for 2D RGB images.

If you want to acutally compare you would have to train my version from scratch.

[netw0rkf10w]

That was my first guess, but then I realized that in my test, sadeepj's CRF also did not have trained weights either, yet still produced good results. Please keep in mind that what I loaded were not CRF weights at all, but only CNN weights (without CRF).

The major difference I can see (apart the permutohedral lattice implementation), is the initialization:

Your code:

self.spatial_ker_weights = self.add_weight(name='spatial_ker_weights',
                                                   shape=(self.num_classes,),
                                                   initializer=tf.initializers.truncated_normal(mean=0, stddev=0.1),
                                                   trainable=True)

self.spatial_ker_weights = tf.diag(self.spatial_ker_weights)

self.bilateral_ker_weights = self.add_weight(name='bilateral_ker_weights',
                                             shape=(self.num_classes,),
                                             initializer=tf.initializers.truncated_normal(mean=0, stddev=0.1),
                                             trainable=True)
self.bilateral_ker_weights = tf.diag(self.bilateral_ker_weights)

self.compatibility_matrix = self.add_weight(name='compatibility_matrix',
                                            shape=(self.num_classes, self.num_classes),
                                            initializer=tf.initializers.truncated_normal(mean=0, stddev=0.1),
                                            trainable=True)

Their code:

self.spatial_ker_weights = self.add_weight(name='spatial_ker_weights',
                                           shape=(self.num_classes, self.num_classes),
                                           initializer=_diagonal_initializer,
                                           trainable=True)

self.bilateral_ker_weights = self.add_weight(name='bilateral_ker_weights',
                                             shape=(self.num_classes, self.num_classes),
                                             initializer=_diagonal_initializer,
                                             trainable=True)

self.compatibility_matrix = self.add_weight(name='compatibility_matrix',
                                            shape=(self.num_classes, self.num_classes),
                                            initializer=_potts_model_initializer,
                                            trainable=True)

where

def _diagonal_initializer(shape):
    return np.eye(shape[0], shape[1], dtype=np.float32)

def _potts_model_initializer(shape):
    return -1 * _diagonal_initializer(shape)

...

Sorry @MiguelMonteiro for the delay of this reply. I was about to submit the above content but then I thought it would worth trying the same initialization for both implementations. So I changed the initialization in your implementation to obtain the same as sadeepj's, and this took me some time to test.

The results are more meaningful now. However, they are still worse than sadeepj's CPU (segmentation could not capture well the edges), although the weights of the CRFs are the same.

[MiguelMonteiro]

Where did you get the pretrained weights from? If they are the ones sadeepj made available then they include the CRF's weights and you they are pretrained and not random at all.

[netw0rkf10w]

It's a bit tricky: I changed the name of the CRF layer in their code (e.g. from crfrnn to crfrnn_cpu) and then loaded the weights with by_name=True.

[MiguelMonteiro]

Can you manually check that they are sampled from the random distribution of the initialisation or if they have been trained?

[netw0rkf10w]

Ooooooooops... You were right!! I didn't notice that when doing some tests last night I changed the layer name from 'crfrnn_cpu' back to 'crfrnn' (which made sadeepj's CRF read their trained CRF weights). Using 'crfrnn_cpu' I obtained almost identical results as your implementation (with only some minor differences here and there). This is great!

To be able to confirm that your implementation works as expected for 2D segmentation, I will train it on Pascal VOC dataset. I will get back in a few days.

Thank you very much again for the helpful discussion!

[MiguelMonteiro]

No problem, also thinks for doing this. If you manage to get some nice trained 2D RGB weights for my version of the model let me know. I can add you as contributor to this repo if you are willing to publicly share those weights.

[netw0rkf10w]

Yes I will certainly share the trained weights. There are also some modifications in the code so I will create a pull request. Planned for Saturday or Sunday.

[gyheart]

Yes I will certainly share the trained weights. There are also some modifications in the code so I will create a pull request. Planned for Saturday or Sunday.

Can you share your trainings code about FCN+CRF of Miguel's and sadeepj's . My code can not get good performance, but I can't find error in my code. My email is gyimage@yeah.net. Thanks.

[gyheart]

Hi! Thanks for the quick response. I have been working on applying a crf-rnn layer to my network which works on single instance images (The image contains only one object and some background). Initially, I integrated the crf-rnn-keras from here. The resulting segmentations showed contours and strong edges from the image.

I tried yours today and I couldn't find any sign that it took the reference image into account. Should the images in your case be normalized between 0 and 255 or 0 and 1? From your examples it seems like it doesn't matter. I also tried out your theta_alpha, theta_beta, theta_gamma but that didn't make any difference in my case.

Can you share your trainings code about FCN+CRF of Miguel's and sadeepj's . My code can not get good performance, but I can't find error in my code. My email is gyimage@yeah.net. Thanks.

[netw0rkf10w]

Sorry I have been too busy to work on the training of @MiguelMonteiro's layer. I will push my code for using this layer next week. You can easily find Keras training code for semantic segmentation on Google (that's what I'm going to do anyway).

[gyheart]

Sorry I have been too busy to work on the training of @MiguelMonteiro's layer. I will push my code for using this layer next week. You can easily find Keras training code for semantic segmentation on Google (that's what I'm going to do anyway).

Sorry to bother you, the code is available now?

[netw0rkf10w]

@gyheart If it's that urgent for you then please make some effort to do it yourself and if you have questions then post them here to get help. I have been too busy the last days sorry...

[gyheart]

@gyheart If it's that urgent for you then please make some effort to do it yourself and if you have questions then post them here to get help. I have been too busy the last days sorry...

I am sorry. These days I also try to train this layer.

[gyheart]

@gyheart If it's that urgent for you then please make some effort to do it yourself and if you have questions then post them here to get help. I have been too busy the last days sorry...

How did you set the parameters like thetas when you train this layer? Thanks.

[netw0rkf10w]

Finally I will have sometime tonight (Europe time) to work on this. Stay tuned for the pull request. Sorry to @MiguelMonteiro for the delay.

[MiguelMonteiro]

No problem, we all have other work to deal with :)