Suggestion for modification on dice loss function

jis478 commented 5 years ago

I was working on an imbalanced problem with your code for a week and the results got me frustrated with low accuracy. However, after I did some work as below the accuracy has significantly improved so I'm wondering whether these approaches are worth discussing here.

Loss function I did modify the dice loss function and it led to an improved accuracy for my imbalanced problem. As far as I'm concerned the dice loss function in the current code involves all the dimensions (even including background dimension) which cause a learning issue for any imbalanced problem. My thought was, it should only consider target class dimension when calculating intersection and union areas. With this modification I resolved accuracy my issue and I also guess this could be possibly an answer to the other predicting-only-black(background) issues as well.
tf.data and tf.Image API I also replaced some of the current data provider codes with Tensorflow tf.data and tf.image API and also found improvements in speed and accuracy.

Thanks,

Regards Mark

jakeret commented 5 years ago

Hi Mark thanks for the input. The dice loss has been bothering me for quite a while. Would you mind to share it (possibly in a Pull Request) so that I can have a look at your improvements?

I haven't worked yet with the two API's. Could you elaborate what you changed and how it improved things?

Best J

jis478 commented 5 years ago

Hi Jakeret, could you please check the below link? I'm new to the github so I'm not very familiar with Pull / Push stuff. Instead I've organized all of my thoughts on my page first. If you think it is worth going forward then please let me know.

https://github.com/jis478/Tensorflow/blob/master/Unet_modified

Regards Mark

jakeret commented 5 years ago

No worries. This looks quite interesting. I had implemented a very similar version of the dice coefficient as you did. However in my case the network struggled to learn anything not even the simple toy problem. This got me quite confused.

As for the two API's. What is your experience, by how much did the performance profit by using it?

jis478 commented 5 years ago

Thanks for the reply. Can you provide me with your imbalanced toy dataset? I will get back to you with the speed & accuracy test result as soon as possible.

jakeret commented 5 years ago

I was just using the toy example of the package (https://github.com/jakeret/tf_unet/blob/master/scripts/launcher.py) its not imbalance but in principle the network should be able to learn the segmentation

jis478 commented 5 years ago

Hi jakeret, sorry for my late reply. I've updated my github page so you can see how the modified code performs better than the original code.

I think this loss issue causes to learn about only colors or simple shapes, not semantic features which is what U-net is originally targeting with.

Modified version example https://github.com/jis478/Tensorflow/blob/master/Unet_modified/example/membrane/code/Execution_membrane.ipynb (Please be aware of that the ground truth image is not properly displayed in the output so I added another cell to display the corresponding ground truth image)
Original version (your code) example https://github.com/jis478/Tensorflow/blob/master/Unet_modified/example/membrane/code/Unet_original_execution.ipynb

jis478 commented 5 years ago

I've updated the result, so please have a look at this:

https://github.com/jis478/Tensorflow/tree/master/Unet_modified/example/membrane/code/Unet_modified_execution.ipynb

jakeret commented 5 years ago

Wow, the results in your notebook look very good. I transfered your implementation into the tf_unet codebase and ran the toy example with it. However, then the network is not learning anything i.e. I get "all black" predictions.

Wondering if I missed something. Have you tried that as well?

jakeret / tf_unet

Suggestion for modification on dice loss function #243