All predcition are road

hszhao / ICNet

ICNet for Real-Time Semantic Segmentation on High-Resolution Images, ECCV2018

https://hszhao.github.io/projects/icnet

605 stars 199 forks source link

All predcition are road #5

Closed JalongMa closed 7 years ago

JalongMa commented 7 years ago

Hi, Hengshuang! This is a valid work for practical application. I've run your code. But something may be wrong. The predictions of network are all road, i.e. every element of 'imPred' is one. I can not locate the problem. I tested many images and got same results. What's wrong with that? thx

revilokeb commented 7 years ago

@JalongMa Strangely I am also seeing only 1 in imPred (in eval_sub.m). net.forward({input_data}) is giving a 2049x1025x19 tensor which has got max value always in the first dimension, i.e. at [:,:,1]. Same for both pre-trained models. Since I have checked that the image that I have have loaded is valid I am a bit puzzled what is going on. Have you solved the issue?

JalongMa commented 7 years ago

@revilokeb I only got the right or reasonable predictions under CPU mode. You can try it.

revilokeb commented 7 years ago

@JalongMa ok, thank you. However, I have reinvestigated the issue and I was having the following problem which resulted in me obtaining only 1s in imPred:

my hardware setup is a machine with one Pascal and two Maxwell GPUs
normally I am running Ubuntu 16 and CUDA 8 + cuDNN 5.1, but I have got another setup on that machine with Ubuntu 14, CUDA 7.5 which I used for ICNet (I removed cuDNN 5.1 and replaced with cuDNN 4)
With that Ubuntu 14 / CUDA 7.5 setup I was running inference on the Pascal GPU (which happened to have GPU ID = 0). But Pascal GPUs require CUDA 8 to work properly.
So I just changed GPU ID to one of the Maxwell GPUs and it is now working flawlessly on GPU

Maybe you are also running CUDA 7.5 on Pascal?

JalongMa commented 7 years ago

@revilokeb As you said, I ran this with CUDA 7.5 on Pascal. Maybe there are something different between Pascal and Maxwell. If you got only 1s under GPU mode, that's inconsistent with claims in the paper. Anyway, authors don't provide training scripts. This really bothers me....

revilokeb commented 7 years ago

@JalongMa As I said, on my Titan X Maxwell the code is running perfectly. I am assuming that they have performed their experiments also on Titan X Maxwell (see their paper https://arxiv.org/pdf/1704.08545.pdf, section 6) and not Pascal. It is known that CUDA 7.5 does not neccessarily run without issues on Pascal, e.g. look at https://devtalk.nvidia.com/default/topic/955165/cuda-setup-and-installation/pascal-titan-x-with-cuda-7-5/ and http://docs.nvidia.com/cuda/pascal-compatibility-guide/index.html#axzz4lCdXo164

JalongMa commented 7 years ago

@revilokeb OK, thank you. I'll try out on Maxwell TITAN. Good luck !