meetps / pytorch-semseg

Semantic Segmentation Architectures Implemented in PyTorch
https://meetshah.dev/semantic-segmentation/deep-learning/pytorch/visdom/2017/06/01/semantic-segmentation-over-the-years.html
MIT License
3.39k stars 798 forks source link

Doesn't train actually. #31

Closed chichivica closed 6 years ago

chichivica commented 7 years ago

Many thanks for great opensource implementation of the semantic segmentation in pytorch ever!

I'm trying to proceed through training 'segnet' model on 'pascal' dataset. What I've done: 1) installed pytorch: 0.2.0_4 and python: 2.7.13 2) downloaded VOCtrainval_11-May-2012.tar from http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#Development%20Kit 3) downloaded "Semantic Boundaries Dataset and Benchmark" from http://home.bharathh.info/pubs/codes/SBD/download.html 4) as stated in Readme, extracted and pointed to them in config.json file 5) started training process

after completion, generated 100 segnet_pascal1%2d.pkl files

6) So, after that I'm trying to test newly trained model on the simple pictures:

python test.py --model_path segnet_pascal_1_99.pkl --dataset pascal --img_path 2007_000033.jpg --out_path result_33.jpg

alt text

But result is quite wrong: alt text

alt text

for some reasons, output resolution differ and segmentation was not produced correctly.

Could you please give me some advises what I'm doing wrong?

alt text

Many thanks, Ivan

chichivica commented 7 years ago

In addition, trained FCN8S, alt text alt text

meetps commented 7 years ago

Refer #20 , Try training fcn-8s without polynomial learning rate. Should work fine.

chichivica commented 7 years ago

@meetshah1995 trained SEGNET with disabled polynomial learning rate (commented out two lines):

iter = len(trainloader)*epoch + i

poly_lr_scheduler(optimizer, args.l_rate, iter)

as well I had to change here like this:

rgb = np.zeros((temp.shape[1], temp.shape[2], 3))

It may be because I'm using python3, or because you previously made a conversion NHWC -> NCWH. I don't know. Have you ever tested Segnet architecture?

So, now I'm waiting util fcn-8s get trained to see results without polynomial learning rate.

chichivica commented 7 years ago

At least fcn-8s got trained. It took approximately 3 day on nvidia 1080.

XavierCHEN34 commented 6 years ago

hi, why does your results looks much worse than the examples?

meetps commented 6 years ago

PolyLR doesn't seem to work with FCN. FCN-8s in default settings work fine, with mIoU > 60

HelloAlone commented 6 years ago

hi, @chichivica if you want to train SegNet, you should set batch_size and l_rate, because of the BN layer. For example , you can set batch_size=16 and l_rate=1e-4.

Galto2000 commented 6 years ago

I am getting the same (bad) results as @chichivica, with SegNet, even with the settings @HelloAlone suggested. My GTX1080Ti just could not handle a batch size of 16, so I used a batch size of 14 instead.

Also, the noise on my training curves increases significantly over time.

newplot 1

xdsonglinliu commented 5 years ago

@chichivica @Galto2000 Thank you for your analysis. I meet the same problem. Have you finally trained the SegNet network successfully? or Is there any other better SegNet implementation based on pytorch?