experiment results (continue updating)

[yzbx]

miou for cityscapes val dataset

repositor	net name	note	miou
deeplab	deeplab_base	-	0.59
deeplab	deeplab_edge	-	0.57
torchseg	pspnet	-	0.50

offical result

net_name	backbone	dataset	note	miou
pspnet	resnet101	voc	-	82.6
pspnet	resnet101	voc	coco	85.4
pspnet	resnet101	cityscape	-	78.4
pspnet	resnet101	cityscape	coarse	80.2
pspnet	resnet50	ade20k	ms	42.78
deeplabv3+	xception	voc	-	87.8
deeplabv3+	xception	voc	JFT	89.0

miou for voc val dataset

network	note	miou
deeplab v3+	paper report	0.8339
deeplab v3+	no change	0.837-0.838
deeplab v3+	+branch	~0.838
deeplab v3+	+ms	0.8422
deeplab v3+	+flip	-

[yzbx]

adam vs sgd

pytorch experiment on cityscapes dataset

model	optimizer	learning rate	miou train	miou val
pspnet 100 epoch	adam	0.0001	0.80	0.55
pspnet 100 epoch	sgd	0.01	0.76	0.53
pspnet 200 epoch	sgd	0.01	0.80	0.54

sgd with 200 epoch, but the miou is on 100/200 epoch for poly learning rate schedule adam with 100 epoch

[yzbx]

edge overfit

[yzbx]

adam vs sgd

pytorch experiment on cityscapes dataset

model	optimizer	learning rate	miou train	miou val
pspnet	adam	0.0001	0.80	0.55
pspnet	sgd	0.01	0.76	0.53

sgd with 200 epoch, but the miou is on 100 epoch for poly learning rate schedule adam with 100 epoch

psp_edge miou

pytorch + cityscapes bug: psp_edge not support poly sgd, use adam default

model	epoch	note	date	train	val
psp_edge	200	base	2018/09/01	0.81	0.53
psp_edge	80	base	2018/09/01	0.74	0.51
psp_edge	160	ignore_index	2018/09/02	0.87	0
psp_edge	80	ignore_index	2018/09/02	0.82	0.50
psp_edge	80	poly edge weight	2018/09/03	0.82	0.50
psp_edge	200	poly edge weight	2018/09/03	0.86	0
psp_edge	200	overfit0.5**2	2018/09/03	0.8+	0
psp_edge	200	balance_weight	2018/09/06	0.76	0.50
psp_edge	200	edge_lass_num4	2018/09/06	0.79	0.53
psp_edge	200	ignore_edge	2018/09/08	?	?

deeplab_edge

note	step	date	train	val
edge_class_num4	30k	2018-09-06	0.27	0.40
train_with_eval	30k	2018-09-06	0.24	0.40
ignore_edge_new	30k	2018-09-08	?	?

pytorch with regularization

model	backbone	epoch	date	train	val
pspnet	resnet50	200	2018/09/08	?	?
pspnet	resnet101	200	2018/09/08	?	?

[yzbx]

pytorch experiment summary 2018-09-09

[yzbx]

edge and seg order

edge_seg_order	date	epoch	train	val
edge512_first	2018/09/13	200	0.7534	0.4889
edge512_later	2018/09/11	200	0.7198	0.4633
edge_first	2018/09/11	200	0.6623	0.4671
edge_later	2018/09/10	200	0.6960	0.4621
edge_same	2018/09/09	200	0.7571	0.4629

edge type

edge type	date	epoch	train	val
semantic edge	2018/09/14	200	0.7415	0.5106
canny edge + semantic edge	2018/09/15	200	0.7563	0.5188

merge type

merge type	epoch	train	val
cross merge 0	100	0.7677	0.5169
cross merge 1	100	0.7362	0.5086
cross merge 2	100	0.7568	0.4975
merge_seg	200	0.8558	0.5397

[yzbx]

norm ways

for norm_ways in pytorch cityscapes -1,1 0,1 caffe
do
   python test/pspnet_test.py --batch_size=4 --net_name=pspnet \
   --augmentation=True --learning_rate=0.001 --optimizer=adam \
   --backbone_name=resnet50 --backbone_freeze=False --midnet_scale=15 \
   --upsample_type=bilinear --backbone_pretrained=True --n_epoch=50 \
   --note=${norm_ways} --norm_ways=${norm_ways}
done

experiments result

   train/iou   val/iou        note
0   0.459440  0.403973       caffe
1   0.460660  0.405963     pytorch
2   0.463188  0.416964  cityscapes
3   0.460112  0.400258        -1,1
4   0.460654  0.401508         0,1

learning_rate

for lr in 5e-4 5e-5 2e-4 2e-5
do
    python test/pspnet_test.py --batch_size=4 --net_name=pspnet --backbone_freeze=False \
--midnet_scale=15 --upsample_type=bilinear --backbone_pretrained=True --n_epoch=50 \
--note=cityscapes_$lr --norm_ways=cityscapes --learning_rate=$lr
done

note	learning rate	time	best train/iou	best val/iou
naive_bn4_aug_T	-	2018/08/09	0.763474	0.488097
cityscapes	1e-3	2018/09/29	0.463188	0.416964
cityscapes	1e-4	2018/09/30	0.723264	0.528135
cityscapes_x	2e-4	-	0.687637	0.528059
cityscapes_x	5e-4	-	0.560245	0.477189
cityscapes_x	2e-5	-	0.697064	0.534142
cityscapes_x	5e-5	-	0.714838	0.537532

set lr_mult = False

python test/pspnet_test.py --batch_size=4 --net_name=pspnet --backbone_freeze=False \
--midnet_scale=15 --upsample_type=bilinear --backbone_pretrained=True --n_epoch=50 \
--note=wo_lr_mult --norm_ways=cityscapes --use_lr_mult=False

python test/pspnet_test.py --batch_size=4 --net_name=pspnet --backbone_freeze=False \
--midnet_scale=15 --upsample_type=bilinear --backbone_pretrained=True --n_epoch=50 \
--note=w_lr_mult --norm_ways=cityscapes --use_lr_mult=True

use lr mult	best train miou	best val miou
True	0.718	0.532
False	0.738	0.526

[yzbx]

learning rate

note=cityscapes*

[yzbx]

batch size

note	time	epoch	best train miou	best val miou
nw_cityscpaes	2018-10-09	100	0.788829	0.529173

note	train/iou	val/iou	n_epoch
bn2_wo_batchnorm	0.750111	0.527493	100
bn4_wo_batchnorm	0.772401	0.521195	100
bs2_wo_bn_city	0.752197	0.523592	100

[yzbx]

norm ways

	train/iou	val/iou	note	n_epoch
0	0.822187	0.523619	bn2_sgd200_nwpytorch	200
1	0.842374	0.544636	bn2_sgd200_nwcity	200
2	0.847756	0.55307	bn2_sgd200_nwcaffe	200

	train/iou	val/iou	note	n_epoch
3	0.659751	0.489004	bn12_nw_pytorch	30
1	0.662099	0.492727	bn12_nw_caffe	30
5	0.657091	0.493804	bn12_nw_pytorch	30
2	0.661716	0.50363	bn12_nw_caffe	30
0	0.663704	0.516348	bn12_nw_caffe	30
4	0.660641	0.524681	bn12_nw_pytorch	30

[yzbx]

bugs in psp_edge

view hyeropt 016 for detail for edge weight = 0.1, 0.2, 0.5, 1.0 max miou = 0.03 compare to 0.38 for pspnet without edge in the same condition

dataset size, note	train/iou	val/iou
(320, 'use_bn_017')	0.788159	0.381482
(320, 'wo_bn_015')	0.77847	0.383078
(320, 'wo_bn_016')	0.02894	0.031192

[yzbx]

2018-11-01 - 2018-11-09

[yzbx]

2018-09-01 - 2018-09-19

[yzbx]

2018-10-01 - 2018-10-19

[yzbx]

2018-09-20 - 2018-09-31

[yzbx]

2018-10-20 - 2018-10-31