search

shekkizh / FCN.tensorflow

Tensorflow implementation of Fully Convolutional Networks for Semantic Segmentation (http://fcn.berkeleyvision.org)
MIT License
1.25k stars 527 forks source link

Getting blank predicted images #54

Closed NitishMutha closed 6 years ago

NitishMutha commented 7 years ago

@shekkizh I took exact code from the git repo and trained without any changes on Titan x for 100000 iterations with default batch size. But on visualizing after completing the training, all i could see is full black images in prediction.

What could possibly go wrong?

NitishMutha commented 7 years ago

image

anthonyjules commented 7 years ago

I have the same result. Any help would be greatly appreciated.

himsR commented 7 years ago

Did you guys got it working?

anthonyjules commented 7 years ago

I haven't gotten it working yet. It seems like the net is just learning a black background. The prediction image starts off with some random noise (expected) and converges to pure black.

I am running this with no changes to the source (same as NitishMutha): Python version: 2.7.12 TensorFlow version: 1.2.1

Can anyone who has gotten it to work share what versions of Python & TF they used?

CLegendBuptSun commented 7 years ago

I have the same result. Had anyone tested the image（the visualization ）correctly？I read the code top to the bottom，but i didnot find the reason（or bug）

dubvulture commented 7 years ago

Did you try lowering the learning rate? (maybe to 1e-5)

himsR commented 7 years ago

@dubvulture I tried .00005 learn rate and batch-size 10. Nothing helps. Did it work for you?

dubvulture commented 7 years ago

I ran a tweaked version of this repo on a different task (only two classes).

So yeah, meddling with these parameters might be the answer to your problems. (or wait for the "official" ones)

aloyschen commented 7 years ago

I have the same question, the predict images all are blank

cdathuraliya commented 7 years ago

@NitishMutha Can you please check whether you get same result for visualizing? In my case I get black images in TensorBoard but actual predictions look OK. I have used this implementation for few other datasets too.

lunersonguo commented 7 years ago

I've got the same problem ,and tried different lr and bs, but still it does not work for me. It seems that training dataset is very imbalanced ( most of them are backgrounds). Does anyone get any solution? or How can I get rid of the background class for learning? Thanks!

shekkizh commented 6 years ago

Possible fix #77

varungupta31 commented 2 years ago

Hi @dubvulture, I'm training on custom data with 2 classes as well, foreground and background. I'm getting results like:

Step: 0, Train_loss:14.583, Train_Set_accuracy:0.375, Train_Set_IoU: 0.362297
Step: 100, Train_loss:0.0970308, Train_Set_accuracy:0.375, Train_Set_IoU: 0.372662
Step: 300, Train_loss:0.0732907, Train_Set_accuracy:0.375, Train_Set_IoU: 0.367829
Step: 650, Train_loss:0.103766, Train_Set_accuracy:0.375, Train_Set_IoU: 0.365488
Step: 700, Train_loss:0.102962, Train_Set_accuracy:0.375, Train_Set_IoU: 0.368576

a. When I call np.unique on my predictions, I get only [0]. Thus, my predictions are only having black pixels. b. Mean accuracy is constant at 0.375. Earlier I thought that my pred_annotations is being calculated on a fixed data (which isn't updating), but the IoU is being calculated on the same data, which is varying, and thus the data/model is indeed updating.

I'm using a batch size of 2, and a learning rate of 1e5+1. I have only run the code for 700 steps, am I to early to judge anything, or there seems to be an issue I should check and correct before continuing?

Kindly help me out, Thanks.