search

ISCAS007 / torchseg

use pytorch to do image semantic segmentation
GNU General Public License v3.0
7 stars 1 forks source link

bayes optimizaton + pytorch (000-027) #21

Closed yzbx closed 5 years ago

yzbx commented 6 years ago

tensor * numpy

# tensor * numpy (okay)
# numpy * tensor (failed)
loss = loss + l2_reg * l2_loss + l1_reg * l1_loss (failed)
loss = loss + l2_loss * l2_reg + l1_loss * l1_reg
yzbx commented 6 years ago

a abnormal bug for pytorch + bayes optimization

Traceback (most recent call last):                                                      File "test/pspnet_test.py", line 288, in <module>
    hyperopt.tpe()
  File "/home/yzbx/git/torchseg/utils/model_hyperopt.py", line 97, in bayes
    bo.maximize(init_points=1,n_iter=1,kappa=2)
  File "/home/yzbx/bin/miniconda3/envs/new/lib/python3.6/site-packages/bayes_opt/bayesian_optimization.py", line 259, in maximize
    **self._acqkw)
  File "/home/yzbx/bin/miniconda3/envs/new/lib/python3.6/site-packages/bayes_opt/helpers.py", line 63, in acq_max
    if max_acq is None or -res.fun[0] >= max_acq:
IndexError: too many indices for array

how to fixed!

bo=bayesopt(target,{'base_lr':[0.01,1e-4],'l1_reg':[1e-3,1e-7],'l2_reg':[1e-3,1e-7]})
-->
bo=bayesopt(target,{'base_lr':[1e-4,0.01],'l1_reg':[1e-7,1e-3],'l2_reg':[1e-7,1e-3]})
yzbx commented 6 years ago

skopt

val_miou* use part of dataset, vgg16 val_miou use full of dataset

note base_lr l1_reg l2_reg val_miou* val_miou
skopt001 1.53e-3 2.38e-4 2.70e-4 0.234 0.448 (resnet50)
skopt002 0.01 5.57e-7 9.29e-4 0.231 0.339(vgg16)

val_miou * use part of dataset resnet50 val_miou use full of dataset, resnet50

note base_lr l1_reg l2_reg val_miou* val_miou
skopt003 1.977e-3 1e-07 1e-3 0.2307 0.550
yzbx commented 6 years ago

hyperopt

python test/pspnet_test.py --test=hyperopt --upsample_type=bilinear \
--n_epoch=100 --note=hyperopt --batch_size=4 \
--backbone_name=vgg16 --hyperopt=skopt --midnet_scale=15

result

    base_lr l1_reg  l2_reg  val_miou
0   0.0015335192657991573   0.00023826650493636675  0.0002702604344019315   0.2314655848682947
1   0.004948840736375756    3.1207198770106414e-05  3.447679044520655e-06   0.01981100924905993
2   0.00039361280014637103  1.6859780231705852e-07  1.232041050356271e-06   0.4465743655424462
3   0.000902256947026915    0.00017728619337514944  8.315889109899302e-06   0.25033817567840694
4   0.0006103368503860312   0.00022096070868729594  2.2365810539197546e-06  0.2599305062777417
5   0.001978535031094702    2.9714344857244974e-06  0.0006739390723749778   0.24770741187457382
6   0.00019085412933888727  0.00030223797682839184  7.842096480821407e-06   0.2715754248462153
7   0.0039978040531536176   1.2075633351351962e-05  5.194193452752142e-05   0.01981100924905993
8   0.002762264801799497    2.1285270235317302e-05  1.4108892283675816e-05  0.24248481393574806
9   0.003290267810262663    2.6523475351677293e-07  7.841545691824227e-06   0.26719292869172623
10  0.0006089438284652259   2.0871343856134564e-06  6.729574270765496e-05   0.40287504064245033
11  0.00950145945711886 8.213526398577149e-07   5.0703356181525875e-05  0.01981100924905993
12  0.0032623571070962186   1.1498058081231439e-06  3.8998973599135995e-06  0.2723520510385146
13  0.01    7.32570216156694e-05    0.00021767079279222412  0.01981100924905993
14  0.01    0.001   1.4744117247909384e-07  0.01981100924905993

hyperopt002

search [base_lr, l1_reg, l2_reg]

python test/pspnet_test.py --test=hyperopt --upsample_type=bilinear \
--n_epoch=200 --optimizer=sgd --use_reg=True --note=hyperopt002 \
--batch_size=4 --backbone_name=vgg16 --hyperopt=skopt \
--midnet_scale=15 --dataset_use_part=True --backbone_pretrained=True

result

minimize score=0.7602 minimize param [0.01, 1e-07, 1e-07]

note

remove l1_reg, fix base_lr, replace sgd with adam, change backbone to resnet50 ?

target

find the best l2_reg for pytorch

yzbx commented 6 years ago

hyperopt003

python test/pspnet_test.py --test=hyperopt --upsample_type=bilinear \
--n_epoch=200 --optimizer=sgd --use_reg=True --note=hyperopt003 \
--batch_size=4 --backbone_name=resnet50 --hyperopt=skopt \
--midnet_scale=15 --dataset_use_part=True --backbone_pretrained=True

result

val_miou base_lr l1_reg l2_reg
0.23066201620876672 0.00197715035086791719 0.00000010000000000000 0.00100000000000000002
0.23054786318734086 0.00193028144481249223 0.00000010000000000000 0.00100000000000000002
0.22424590394454649 0.00169139248460557800 0.00000010000000000000 0.00100000000000000002
0.22402201376838249 0.00159572070583458773 0.00000015269113619765 0.00097246785636114256
0.22255296318121179 0.00189727557050649309 0.00000010000000000000 0.00100000000000000002
0.22174483433707845 0.00134330934533122260 0.00000017149331262640 0.00099488818508320048

note

no stable conclusion

target

find the best lr + l1_reg + l2_reg

yzbx commented 6 years ago

hyperopt004

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=True \
--test=hyperopt --note=hyperopt004 --hyperopt=loop

result

screenshot from 2018-09-23 14-53-44

max******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.737600  0.222334
0,1          0.733858  0.206601
caffe        0.737186  0.214802
cityscapes   0.743215  0.231846
pytorch      0.730023  0.208678
mean******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.731978  0.186179
0,1          0.728715  0.189355
caffe        0.711750  0.191525
cityscapes   0.727899  0.200267
pytorch      0.727013  0.187163

note

caffe are more stable, var is small cityscapes have the best result, mean and max is best.

target

find the best normalization method for cityscapes dataset

pspnet

python test/pspnet_test.py --batch_size=4 --backbone_pretrained=True \
--midnet_scale=15 --upsample_type=bilinear --norm_ways=cityscapes \
--note=new_norm_ways

best val iou is 0.540 

for norm_ways in pytorch cityscapes -1,1 0,1 caffe
do
   python test/pspnet_test.py --batch_size=4 --net_name=pspnet \  
   --augmentation=True --learning_rate=0.001 --optimizer=adam \
   --backbone_name=resnet50 --backbone_freeze=False --midnet_scale=15 \
   --upsample_type=bilinear --backbone_pretrained=True --n_epoch=50 \
   --note=${norm_ways} --norm_ways=${norm_ways}
done
yzbx commented 6 years ago

hyperopt005

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=True \
--test=hyperopt --note=hyperopt005 --hyperopt=loop

target

find the best l2 reg

result

screenshot from 2018-09-24 13-36-54

max******************************
        train/iou   val/iou
l2_reg                     
0.0001   0.745121  0.221688
0.001    0.725888  0.221025
0.01     0.711914  0.220541
0.1      0.533035  0.197450
1e-05    0.738100  0.209281
mean******************************
        train/iou   val/iou
l2_reg                     
0.0001   0.727874  0.191264
0.001    0.721769  0.185075
0.01     0.694604  0.194846
0.1      0.516797  0.181090
1e-05    0.729217  0.187572

note

理论上,l2_reg 与 loss 构成的函数应该满足一阶导数单调,从而使得其在区间有且仅有一个最大值,但上述结果显然不满足.

yzbx commented 6 years ago

hyperopt006

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=320 \
--norm_ways=caffe --hyperkey=model.l2_reg \
--test=hyperopt --note=hyperopt006 --hyperopt=loop

target

use caffe norm ways and set dataset size from 32 to 320 to decrease random noise

result

max******************************
        train/iou   val/iou
l2_reg                     
0.0001   0.797486  0.377943
0.001    0.783965  0.381637
0.01     0.658299  0.366019
0.1      0.349307  0.254865
1e-05    0.788434  0.378732
mean******************************
        train/iou   val/iou
l2_reg                     
0.0001   0.785462  0.370618
0.001    0.777588  0.374658
0.01     0.651019  0.349297
0.1      0.340829  0.241365
1e-05    0.782067  0.373603
yzbx commented 6 years ago

hyperopt007

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=320 \
--hyperkey=dataset.norm_ways --test=hyperopt \
--note=hyperopt007 --hyperopt=loop

target

set dataset size from 32 to 320

result

max******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.796653  0.377777
0,1          0.797381  0.381825
caffe        0.789363  0.379361
cityscapes   0.793626  0.392324
pytorch      0.792334  0.383774
mean******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.787498  0.364685
0,1          0.783770  0.368864
caffe        0.782131  0.371920
cityscapes   0.785335  0.371775
pytorch      0.782552  0.368613
yzbx commented 6 years ago

hyperopt008 009 010

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=32 \
--hyperkey=dataset.norm_ways --test=hyperopt \
--note=hyperopt008 --hyperopt=loop \
--augmentation=False

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=320 \
--hyperkey=dataset.norm_ways --test=hyperopt \
--note=hyperopt008 --hyperopt=loop \
--augmentation=True --hyperopt_calls=25

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=32 \
--hyperkey=dataset.norm_ways --test=hyperopt \
--note=hyperopt009 --hyperopt=loop \
--backbone_freeze=True

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=False --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=32 \
--hyperkey=dataset.norm_ways --test=hyperopt \
--note=hyperopt010 --hyperopt=loop

target

set dataset size to 32 008 remove augmentation, target to solve overfit 009 freeze backbone (use onle 2G GPU, otherwise 8G) 010 without pretrained backbone

conclusion

008 current augmentation has little effect on iou 009 backbone_freeze be True better 010 backbone_pretrained be True better

results

max hyperopt008 norm_ways ******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.734129  0.223043
0,1          0.743829  0.209925
caffe        0.740234  0.203969
cityscapes   0.734632  0.216859
pytorch      0.737144  0.216049
mean hyperopt008 norm_ways ******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.725734  0.190831
0,1          0.729589  0.193581
caffe        0.731093  0.184074
cityscapes   0.725789  0.199930
pytorch      0.726876  0.189584
max hyperopt009 norm_ways ******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.715410  0.204841
0,1          0.716581  0.205622
caffe        0.714231  0.202703
cityscapes   0.718410  0.207391
pytorch      0.719986  0.215195
mean hyperopt009 norm_ways ******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.706099  0.187279
0,1          0.709263  0.185067
caffe        0.710132  0.183899
cityscapes   0.706856  0.178887
pytorch      0.706030  0.184660
max hyperopt010 norm_ways ******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.676492  0.152615
0,1          0.679592  0.154941
caffe        0.684939  0.150943
cityscapes   0.681049  0.126306
pytorch      0.680471  0.150084
mean hyperopt010 norm_ways ******************************
            train/iou   val/iou
norm_ways                      
-1,1         0.656932  0.132506
0,1          0.655741  0.137881
caffe        0.662873  0.133385
cityscapes   0.654435  0.121993
pytorch      0.653812  0.130335
yzbx commented 6 years ago

hyperopt011

use_lr_mult + backbone_pretrained

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=32 \
--hyperkey=model.use_lr_mult,model.backbone_pretrained --test=hyperopt \
--note=hyperopt011 --hyperopt=loop

conclusion

use_lr_mult=False will better backbone_pretrained=True will better

hyperopt012 013

python test/pspnet_test.py --batch_size=4 --use_reg=True --backbone_pretrained=True \
--midnet_scale=15 --upsample_type=bilinear --dataset_use_part=320 \
--hyperkey=model.learning_rate --test=hyperopt --note=hyperopt012 --hyperopt=skopt

python test/pspnet_test.py --batch_size=4 --use_reg=True --backbone_pretrained=True \
--midnet_scale=15 --upsample_type=bilinear --dataset_use_part=640 \
--hyperkey=model.learning_rate --test=hyperopt --note=hyperopt013 --hyperopt=skopt

learning_rate around 1e-4 will be better for adam

    train/iou   val/iou         note  learning_rate
16   0.357555  0.273255  hyperopt012      32.902678
18   0.394275  0.297194  hyperopt012      27.622648
33   0.538827  0.309125  hyperopt012      15.335193
37   0.695417  0.348852  hyperopt012       9.022569
24   0.733858  0.349068  hyperopt012       7.908760
47   0.733766  0.351638  hyperopt012       6.103369
5    0.755240  0.357506  hyperopt012       3.792917
6    0.786611  0.384560  hyperopt012       1.000000
35   0.784682  0.385403  hyperopt012       1.000000
    train/iou   val/iou         note  learning_rate
32   0.302732  0.253993  hyperopt013      49.488407
16   0.325483  0.260770  hyperopt013      39.978041
41   0.370776  0.294552  hyperopt013      27.622648
10   0.352339  0.301990  hyperopt013      32.902678
23   0.393495  0.333704  hyperopt013      19.785350
15   0.508707  0.343164  hyperopt013      15.335193
43   0.666636  0.385745  hyperopt013       9.022569
4    0.739569  0.398805  hyperopt013       6.103369
38   0.715220  0.401254  hyperopt013       7.534755
37   0.769597  0.413027  hyperopt013       3.327895
45   0.796475  0.418749  hyperopt013       3.936128
40   0.815788  0.420533  hyperopt013       2.808254
0    0.812492  0.451280  hyperopt013       1.003466
6    0.812712  0.454330  hyperopt013       1.004154
yzbx commented 6 years ago

hyperopt014

note,learning_rate mean train/iou mean val/iou n_epoch
('hyperopt014', '1e-05') 0.731833 0.323429 100
('hyperopt014', '2e-05') 0.747784 0.334623 100
('hyperopt014', 0.001) 0.684718 0.342896 100
('hyperopt014', '5e-05') 0.758231 0.359602 100
('hyperopt014', 0.0005) 0.756845 0.363612 100
('hyperopt014', 0.0001) 0.782894 0.370021 100
('hyperopt014', 0.0002) 0.799771 0.371424 100
note,learning_rate ('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('hyperopt014', 0.0001) 0.370021 0.00833878 0.382354
('hyperopt014', 0.0002) 0.371424 0.0103945 0.386256
('hyperopt014', 0.0005) 0.363612 0.00266757 0.367989
('hyperopt014', 0.001) 0.342896 0.0136704 0.358228
('hyperopt014', '1e-05') 0.323429 0.0115674 0.340416
('hyperopt014', '2e-05') 0.334623 0.00548159 0.340169
('hyperopt014', '5e-05') 0.359602 0.0102995 0.380692
yzbx commented 6 years ago

015 without batchnorm test for all norm_ways=['caffe','pytorch','cityscapes','-1,1','0,1']

from now on, bn means batch norm, bs means batch size

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=320 \
--hyperkey=dataset.norm_ways --test=hyperopt \
--note=wo_bn_015 --hyperopt=loop
note, norm_ways mean train/iou mean val/iou n_epoch
('wo_bn_015', '-1,1') 0.765514 0.370404 100
('wo_bn_015', 'cityscapes') 0.763883 0.370746 100
('wo_bn_015', 'caffe') 0.763176 0.371277 100
('wo_bn_015', 'pytorch') 0.764656 0.372533 100
('wo_bn_015', '0,1') 0.766019 0.37388 100
note, norm_ways ('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('wo_bn_015', '-1,1') 0.370404 0.00626109 0.383078
('wo_bn_015', '0,1') 0.37388 0.00509405 0.381805
('wo_bn_015', 'caffe') 0.371277 0.00549621 0.381298
('wo_bn_015', 'cityscapes') 0.370746 0.00486433 0.379043
('wo_bn_015', 'pytorch') 0.372533 0.00358674 0.376582

compare 015 with 007, without batchnorm, for mean is better, for max is worse

there is a very bad result for 007, so with batchnorm will better.

test mean iou best iou
007 0.36-0.37 0.37- 0.39
015 0.37 0.37 - 0.38

016 without batchnorm, test for edge base weight=[0.1,0.3,0.5,0.7,0.9]

from now on, bn means batch norm, bs means batch size

python test/pspnet_test.py --dataset_use_part=320 --backbone_pretrained=True \
 --midnet_scale=15 --upsample_type=bilinear --hyperkey=model.edge_base_weight  \
--note=wo_bn_016 --net_name=psp_edge --batch_size=4 --test=hyperopt
note, edge_base_weight mean train/iou mean val/iou n_epoch
('wo_bn_016', 0.2) 0.00962902 0.00962318 100
('wo_bn_016', 1.0) 0.0122471 0.0111641 100
('wo_bn_016', 0.1) 0.0124438 0.0127975 100
('wo_bn_016', 0.5) 0.0148246 0.0145872 100
note, edge_base_weight ('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('wo_bn_016', 0.1) 0.0127975 0.00486456 0.019431
('wo_bn_016', 0.2) 0.00962318 0.00592278 0.0210826
('wo_bn_016', 0.5) 0.0145872 0.00920449 0.031192
('wo_bn_016', 1.0) 0.0111641 0.00574537 0.0198873

017 use_bn=[True,False]

default momentum change from 0.9 to 0.1 from now on

python pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=320 \
--hyperkey=model.use_bn --test=hyperopt \
--note=use_bn_017 --hyperopt=loop --hyperopt_calls=10
train/iou val/iou n_epoch
('use_bn_017', True) 0.6922 0.351351 100
('use_bn_017', False) 0.786587 0.374173 100
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('use_bn_017', False) 0.374173 0.00286126 0.378675
('use_bn_017', True) 0.351351 0.0593905 0.381482
train/iou val/iou note use_bn n_epoch
2 0.228543 0.230323 use_bn_017 True 100
1 0.786149 0.371114 use_bn_017 False 100
8 0.784467 0.372449 use_bn_017 False 100
4 0.78702 0.378675 use_bn_017 False 100
9 0.78334 0.381482 use_bn_017 True 100

018 momentum=[0.1,0.3,0.5,0.7,0.9]

default momentum change from 0.9 to 0.1 from now on

python test/pspnet_test.py --batch_size=4 --use_reg=True \
--backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=320 \
--hyperkey=model.momentum --test=hyperopt \
--note=momentum_018 --hyperopt=loop --hyperopt_calls=25
train/iou val/iou n_epoch
('momentum_018', 0.1) 0.70246 0.351371 100
('momentum_018', 0.5) 0.78483 0.373172 100
('momentum_018', 0.9) 0.786037 0.375285 100
('momentum_018', 0.3) 0.784356 0.376164 100
('momentum_018', 0.7) 0.785318 0.378329 100
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('momentum_018', 0.1) 0.351371 0.0652962 0.381299
('momentum_018', 0.3) 0.376164 0.00563704 0.38015
('momentum_018', 0.5) 0.373172 0.010078 0.382774
('momentum_018', 0.7) 0.378329 0.00609081 0.388035
('momentum_018', 0.9) 0.375285 0.00742035 0.379415
train/iou val/iou note momentum n_epoch
3 0.229106 0.203622 momentum_018 0.1 100
2 0.789433 0.358694 momentum_018 0.5 100
16 0.777101 0.362424 momentum_018 0.5 100
6 0.789024 0.384335 momentum_018 0.7 100
4 0.782291 0.388035 momentum_018 0.7 100
yzbx commented 6 years ago

019 fcn 8, 16 and 32

note upsample_layer midnet_scale
fcn8_019 5 3
fcn16_019 4 5
fcn32_019 3 10 
python test/pspnet_test.py --net_name=fcn --upsample_layer=5 --backbone_name=vgg16 \
--backbone_pretrained=True --note=fcn32_019 --midnet_scale=3 --batch_size=4 \
--dataset_use_part=320 --hyperkey=dataset.norm_ways --test=hyperopt \
--hyperopt=loop --hyperopt_calls=25

best params is******************************
dataset.norm_ways -1,1
val_miou 0.40571112668907283
best score is 0.406******************************
train/iou val/iou n_epoch
('fcn8_019', 3) 0.526633 0.321094 100
('fcn16_019', 4) 0.583918 0.377341 100
('fcn32_019', 5) 0.560772 0.396671 100
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('fcn16_019', 4) 0.377341 0.00258301 0.383401
('fcn32_019', 5) 0.396671 0.00460049 0.405711
('fcn8_019', 3) 0.321094 0.00292513 0.328114
train/iou val/iou note upsample_layer n_epoch
59 0.526188 0.316022 fcn8_019 3 100
61 0.526954 0.328114 fcn8_019 3 100
13 0.584549 0.373556 fcn16_019 4 100
2 0.581644 0.383401 fcn16_019 4 100
27 0.562077 0.388568 fcn32_019 5 100
46 0.562599 0.405711 fcn32_019 5 100

020 fcn32

python test/pspnet_test.py --net_name=fcn --upsample_type=fcn --input_shape=384 \
--backbone_name=vgg16 --backbone_pretrained=True --hyperkey=model.use_bias \
--test=hyperopt --hyperopt=loop --hyperopt_calls=6 --note=fcn32_020 --batch_size=4 \
--upsample_layer=5
train/iou val/iou n_epoch
('fcn32_020', False) 0.512657 0.358855 100
('fcn32_020', True) 0.512293 0.369321 100
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('fcn32_020', False) 0.358855 0.00243141 0.360574
('fcn32_020', True) 0.369321 0.0060562 0.373496
train/iou val/iou note use_bias n_epoch
1 0.510674 0.357136 fcn32_020 False 100
0 0.513492 0.360499 fcn32_020 True 100
5 0.514639 0.360574 fcn32_020 False 100
3 0.512988 0.370236 fcn32_020 True 100
4 0.511617 0.373054 fcn32_020 True 100
2 0.511074 0.373496 fcn32_020 True 100
yzbx commented 5 years ago

use_lr_mult021 vs hyperopt011

python test/pspnet_test.py --batch_size=4 --backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=320 --hyperkey=model.use_lr_mult \
--test=hyperopt --note=use_lr_mult021 --hyperopt=loop --hyperopt_calls=6
train/iou val/iou n_epoch
('use_lr_mult021', False) 0.618404 0.35428 100
('use_lr_mult021', True) 0.579459 0.372767 100
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('use_lr_mult021', False) 0.35428 0.00847738 0.364068
('use_lr_mult021', True) 0.372767 0.00237982 0.37482
train/iou val/iou note use_lr_mult n_epoch
0 0.614281 0.349268 use_lr_mult021 False 100
4 0.624395 0.349505 use_lr_mult021 False 100
1 0.616536 0.364068 use_lr_mult021 False 100
3 0.582392 0.369343 use_lr_mult021 True 100
5 0.572499 0.373212 use_lr_mult021 True 100
2 0.580582 0.373693 use_lr_mult021 True 100
6 0.582364 0.37482 use_lr_mult021 True 100
yzbx commented 5 years ago

lr_mult022

:x: the results may under fit

python test/pspnet_test.py --batch_size=4 --backbone_pretrained=True --midnet_scale=15 \
--upsample_type=bilinear --dataset_use_part=320  --use_lr_mult=True \
--upsample_layer=5 --use_momentum=True \
--hyperkey=model.changed_lr_mult,model.new_lr_mult \
--test=hyperopt --note=lr_mult022 --hyperopt=loop --hyperopt_calls=24
train/iou val/iou n_epoch
('lr_mult022', 10, 10) 0.63345 0.383462 100
('lr_mult022', 2, 20) 0.753019 0.405683 100
('lr_mult022', 1, 10) 0.7587 0.406828 100
('lr_mult022', 1, 20) 0.763231 0.407522 100
('lr_mult022', 5, 10) 0.718829 0.410031 100
('lr_mult022', 5, 20) 0.715609 0.411405 100
('lr_mult022', 2, 10) 0.80022 0.414526 100
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('lr_mult022', 1, 10) 0.406828 0.0161691 0.43087
('lr_mult022', 1, 20) 0.407522 0.00410269 0.411188
('lr_mult022', 2, 10) 0.414526 0.00935819 0.425291
('lr_mult022', 2, 20) 0.405683 0.0095121 0.416596
('lr_mult022', 5, 10) 0.410031 0.0085403 0.419578
('lr_mult022', 5, 20) 0.411405 0.00830431 0.419604
('lr_mult022', 10, 10) 0.383462 0.00378359 0.386138
train/iou val/iou note changed_lr_mult new_lr_mult n_epoch
11 0.593878 0.380787 lr_mult022 10 10 100
16 0.673022 0.386138 lr_mult022 10 10 100
20 0.638207 0.397259 lr_mult022 1 10 100
2 0.806874 0.397357 lr_mult022 1 10 100
17 0.74347 0.399152 lr_mult022 2 20 100
6 0.75651 0.401301 lr_mult022 2 20 100
9 0.794033 0.401824 lr_mult022 1 10 100
14 0.714619 0.402999 lr_mult022 5 20 100
15 0.752931 0.403091 lr_mult022 1 20 100
10 0.708669 0.403119 lr_mult022 5 10 100
8 0.71638 0.407396 lr_mult022 5 10 100
7 0.751457 0.408287 lr_mult022 1 20 100
5 0.792696 0.408331 lr_mult022 2 10 100
18 0.79777 0.409956 lr_mult022 2 10 100
13 0.785305 0.411188 lr_mult022 1 20 100
12 0.710705 0.411611 lr_mult022 5 20 100
4 0.759076 0.416596 lr_mult022 2 20 100
0 0.731436 0.419578 lr_mult022 5 10 100
19 0.721501 0.419604 lr_mult022 5 20 100
3 0.810192 0.425291 lr_mult022 2 10 100
1 0.795687 0.43087 lr_mult022 1 10 100
yzbx commented 5 years ago

upsample_layer023 + vgg16

python test/pspnet_test.py --backbone_pretrained=True --backbone_name=vgg16 \
--upsample_type=bilinear --midnet_scale=15 --batch_size=4 --note=upsample_layer023 \
--test=hyperopt --hyperkey=model.upsample_layer --hyperopt_calls=9 \
--use_momentum=True --dataset_use_part=320
train/iou val/iou n_epoch
('upsample_layer023', 3) 0.600635 0.385072 100
('upsample_layer023', 5) 0.727116 0.422846 100
('upsample_layer023', 4) 0.734326 0.427534 100
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('upsample_layer023', 3) 0.385072 0.00480542 0.389422
('upsample_layer023', 4) 0.427534 0.00789402 0.436232
('upsample_layer023', 5) 0.422846 0.00489393 0.427709
train/iou val/iou note upsample_layer n_epoch
7 0.599051 0.379914 upsample_layer023 3 100
4 0.597684 0.38588 upsample_layer023 3 100
5 0.605171 0.389422 upsample_layer023 3 100
3 0.727067 0.417922 upsample_layer023 5 100
0 0.746137 0.420824 upsample_layer023 4 100
2 0.725659 0.422909 upsample_layer023 5 100
1 0.735279 0.425546 upsample_layer023 4 100
6 0.728623 0.427709 upsample_layer023 5 100
8 0.721564 0.436232 upsample_layer023 4 100
yzbx commented 5 years ago

midnet_scale024/025

voc input size test

for times in 1 2
do
    for midnet_scale in 8 10
    do
        python test/pspnet_test.py --backbone_pretrained=True --backbone_name=vgg19 \
--upsample_type=bilinear --midnet_scale=${midnet_scale} --batch_size=4 --note=midnet_scale024 \
--use_momentum=False --upsample_layer=3 --dataset_name=VOC2012

        python test/pspnet_test.py --backbone_pretrained=True --backbone_name=vgg19 \
--upsample_type=bilinear --midnet_scale=${midnet_scale} --batch_size=4 --note=midnet_scale025 \
--use_momentum=True --upsample_layer=5 --dataset_name=VOC2012
    done
done
train/iou val/iou n_epoch dataset_use_part
('midnet_scale024', 10, 'voc2012') 0.606869 0.341419 100 0
('midnet_scale024', 8, 'voc2012') 0.643355 0.345108 100 0
('midnet_scale024', 8, 'cityscapes') 0.619105 0.511469 100 0
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('midnet_scale024', 8, 'cityscapes') 0.511469 0.00884949 0.517727
('midnet_scale024', 8, 'voc2012') 0.345108 0.00792498 0.349826
('midnet_scale024', 10, 'voc2012') 0.341419 0.000142014 0.341519
train/iou val/iou note midnet_scale name n_epoch dataset_use_part
6 0.640349 0.333286 midnet_scale024 8 voc2012 100 0
4 0.606552 0.341318 midnet_scale024 10 voc2012 100 0
5 0.607186 0.341519 midnet_scale024 10 voc2012 100 0
0 0.647836 0.347907 midnet_scale024 8 voc2012 100 0
7 0.640909 0.349415 midnet_scale024 8 voc2012 100 0
1 0.644325 0.349826 midnet_scale024 8 voc2012 100 0
2 0.619137 0.505212 midnet_scale024 8 cityscapes 100 0
3 0.619073 0.517727 midnet_scale024 8 cityscapes 100 0
train/iou val/iou n_epoch dataset_use_part
('midnet_scale025', 10, 'voc2012') 0.566703 0.238734 100 0
('midnet_scale025', 8, 'voc2012') 0.734451 0.298159 100 0
('midnet_scale025', 8, 'cityscapes') 0.692129 0.544006 100 0
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('midnet_scale025', 8, 'cityscapes') 0.544006 0.00442289 0.547134
('midnet_scale025', 8, 'voc2012') 0.298159 0.0276832 0.324103
('midnet_scale025', 10, 'voc2012') 0.238734 0.0456304 0.270999
train/iou val/iou note midnet_scale name n_epoch dataset_use_part
5 0.427821 0.206468 midnet_scale025 10 voc2012 100 0
0 0.54923 0.259576 midnet_scale025 8 voc2012 100 0
4 0.705586 0.270999 midnet_scale025 10 voc2012 100 0
6 0.804351 0.299119 midnet_scale025 8 voc2012 100 0
1 0.774122 0.309839 midnet_scale025 8 voc2012 100 0
7 0.810102 0.324103 midnet_scale025 8 voc2012 100 0
3 0.695703 0.540879 midnet_scale025 8 cityscapes 100 0
2 0.688555 0.547134 midnet_scale025 8 cityscapes 100 0
yzbx commented 5 years ago

class_weight026/027

class wise weight for semantic segmentation

python test/pspnet_test.py --backbone_pretrained=True --backbone_name=resnet50 \
--upsample_type=bilinear --midnet_scale=15 --batch_size=4 --note=class_weight026 \
--test=hyperopt --hyperkey=model.use_class_weight --hyperopt_calls=6 \
--use_momentum=False --upsample_layer=3 --dataset_use_part=320

python test/pspnet_test.py --backbone_pretrained=True --backbone_name=resnet50 \
--upsample_type=bilinear --midnet_scale=15 --batch_size=4 --note=class_weight027 \
--test=hyperopt --hyperkey=model.use_class_weight --hyperopt_calls=6 \
--use_momentum=True --upsample_layer=5 --dataset_use_part=320
train/iou val/iou n_epoch
('class_weight026', True) 0.5229 0.362163 100
('class_weight026', False) 0.576882 0.375878 100
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('class_weight026', False) 0.375878 0.0103481 0.383294
('class_weight026', True) 0.362163 0.00865791 0.372003
train/iou val/iou note use_class_weight n_epoch
4 0.515585 0.355712 class_weight026 True 100
1 0.532379 0.358775 class_weight026 True 100
3 0.585927 0.364056 class_weight026 False 100
0 0.520735 0.372003 class_weight026 True 100
2 0.568215 0.380284 class_weight026 False 100
5 0.576503 0.383294 class_weight026 False 100
train/iou val/iou n_epoch
('class_weight027', True) 0.613142 0.353979 100
('class_weight027', False) 0.616547 0.380429 100
('val/iou', 'mean') ('val/iou', 'std') ('val/iou', 'amax')
('class_weight027', False) 0.380429 nan 0.380429
('class_weight027', True) 0.353979 0.0163004 0.371405
train/iou val/iou note use_class_weight n_epoch
3 0.606533 0.339105 class_weight027 True 100
0 0.612509 0.351428 class_weight027 True 100
2 0.620383 0.371405 class_weight027 True 100
1 0.616547 0.380429 class_weight027 False 100