search

microsoft / Semi-supervised-learning

A Unified Semi-Supervised Learning Codebase (NeurIPS'22)
https://usb.readthedocs.io
MIT License
1.37k stars 182 forks source link

Performance about Beginning example #73

Closed eshoyuan closed 1 year ago

eshoyuan commented 1 year ago

I run the Beginning_example.ipynb and found the accuracy was only 0.1. I modified the optimization configs according to the comment, but it didn't work.

This is the output.

Epoch: 0
[2023-02-19 06:32:03,336 INFO] confusion matrix
[2023-02-19 06:32:03,337 INFO] [[0.00719161 0.35677718 0.01109215 0.14383228 0.01389566 0.01791809
  0.07557289 0.20075573 0.05741102 0.11555339]
 [0.0442976  0.39304944 0.0150514  0.05347528 0.02300538 0.00709741
  0.12628488 0.12983358 0.04111601 0.16678904]
 [0.00696056 0.41262669 0.00512883 0.08291611 0.02369032 0.01428746
  0.07839785 0.18219563 0.08511418 0.10868238]
 [0.02588839 0.38038833 0.01318842 0.10037856 0.02955184 0.01184516
  0.09842472 0.15337648 0.07681036 0.11014776]
 [0.01193521 0.50468883 0.00815979 0.04859335 0.02094751 0.01181342
  0.05285592 0.1592985  0.09000122 0.09170625]
 [0.02499695 0.41970491 0.00390196 0.07438117 0.02694793 0.01512011
  0.08194123 0.16144373 0.08462383 0.10693818]
 [0.01517005 0.53890384 0.00293614 0.03816981 0.02581356 0.00599462
  0.07805236 0.1225838  0.06435038 0.10802545]
 [0.02513115 0.40856411 0.01207759 0.05904599 0.01402952 0.01390753
  0.05916799 0.20336709 0.10076857 0.10394047]
 [0.00782779 0.30112524 0.01492172 0.11778376 0.02800881 0.01284247
  0.12402153 0.18113992 0.05088063 0.16144814]
 [0.03947849 0.35335689 0.026319   0.09491897 0.0240039  0.01279396
  0.11441452 0.1657122  0.0459364  0.12306568]]
[2023-02-19 06:32:03,339 INFO] evaluation metric
[2023-02-19 06:32:03,339 INFO] acc: 0.0996
[2023-02-19 06:32:03,339 INFO] precision: 0.0900
[2023-02-19 06:32:03,340 INFO] recall: 0.0997
[2023-02-19 06:32:03,340 INFO] f1: 0.0755
model saved: [./saved_models/fixmatch/latest_model.pth](https://vscode-remote+ssh-002dremote-002bzy.vscode-resource.vscode-cdn.net/home/yxyuan/thesis/AutoMTL/saved_models/fixmatch/latest_model.pth)
model saved: [./saved_models/fixmatch/model_best.pth](https://vscode-remote+ssh-002dremote-002bzy.vscode-resource.vscode-cdn.net/home/yxyuan/thesis/AutoMTL/saved_models/fixmatch/model_best.pth)
Epoch: 1
[2023-02-19 06:33:48,335 INFO] confusion matrix
[2023-02-19 06:33:48,336 INFO] [[0.00804486 0.34921989 0.00609459 0.15102389 0.01377377 0.02291565
  0.07874208 0.19246709 0.05143832 0.12627986]
 [0.04123837 0.3726138  0.01199217 0.06069506 0.02227117 0.00599608
  0.13350465 0.13264807 0.04698972 0.17205091]
 [0.00879228 0.39821712 0.00610575 0.08889974 0.02173648 0.0162413
  0.08242765 0.16998413 0.10514104 0.10245451]
 [0.03284894 0.37330565 0.0122115  0.10037856 0.02869703 0.01599707
  0.09170839 0.15533032 0.08352668 0.10599585]
 [0.0140056  0.51613689 0.00840336 0.06199001 0.02289611 0.01071733
  0.04871514 0.14297893 0.09426379 0.07989283]
 [0.0241434  0.41263261 0.00195098 0.0915742  0.0241434  0.01304719
  0.0768199  0.15534691 0.08474576 0.11559566]
 [0.01517005 0.53976022 0.00391485 0.04404208 0.02165402 0.0118669
  0.07120137 0.11707854 0.07413751 0.10117446]
 [0.03098695 0.39929242 0.00902769 0.06490179 0.01512749 0.01280956
  0.07002562 0.18409174 0.11821398 0.09552275]
 [0.00978474 0.30858611 0.00990705 0.13478474 0.02495108 0.01516634
  0.11056751 0.16976517 0.05234834 0.16413894]
 [0.0433776  0.34324357 0.02254173 0.09650299 0.02217619 0.01376873
  0.11319605 0.17131717 0.0456927  0.12818326]]
[2023-02-19 06:33:48,338 INFO] evaluation metric
[2023-02-19 06:33:48,338 INFO] acc: 0.0958
[2023-02-19 06:33:48,338 INFO] precision: 0.0880
[2023-02-19 06:33:48,339 INFO] recall: 0.0959
[2023-02-19 06:33:48,339 INFO] f1: 0.0734
model saved: [./saved_models/fixmatch/latest_model.pth](https://vscode-remote+ssh-002dremote-002bzy.vscode-resource.vscode-cdn.net/home/yxyuan/thesis/AutoMTL/saved_models/fixmatch/latest_model.pth)
Epoch: 2
[2023-02-19 06:35:27,222 INFO] confusion matrix
[2023-02-19 06:35:27,223 INFO] [[0.00499756 0.35031692 0.00792296 0.14553876 0.01572404 0.01487079
  0.07423208 0.21635787 0.05216967 0.11786933]
 [0.04038179 0.39880078 0.01590798 0.05580029 0.02337249 0.00697504
  0.13839941 0.12652961 0.04307391 0.15075869]
 [0.00696056 0.41396996 0.00537306 0.08975455 0.01990475 0.01526438
  0.07058249 0.18671388 0.08291611 0.10856026]
 [0.02075956 0.3963854  0.00818171 0.09720357 0.02381243 0.01697399
  0.08377091 0.15997069 0.08254976 0.11039199]
 [0.01108269 0.51345756 0.00608939 0.05943247 0.01863354 0.00682012
  0.05334308 0.16721471 0.08439898 0.07952746]
 [0.02499695 0.42653335 0.00292647 0.07706377 0.02292403 0.0120717
  0.07779539 0.16717473 0.08279478 0.10571881]
 [0.01920724 0.52263274 0.00587228 0.04685588 0.02361145 0.00990947
  0.06912161 0.13861023 0.05713237 0.10704673]
 [0.02598512 0.40234232 0.01012566 0.06307186 0.01329755 0.0089057
  0.06307186 0.20702696 0.10272051 0.10345248]
 [0.00880626 0.29977984 0.0199364  0.12597847 0.02605186 0.01663405
  0.10518591 0.20303327 0.043909   0.15068493]
 [0.03375168 0.35603753 0.02034848 0.09467528 0.01705861 0.01584014
  0.11989765 0.17533813 0.03777263 0.12927988]]
[2023-02-19 06:35:27,224 INFO] evaluation metric
[2023-02-19 06:35:27,224 INFO] acc: 0.0986
[2023-02-19 06:35:27,225 INFO] precision: 0.0853
[2023-02-19 06:35:27,225 INFO] recall: 0.0986
[2023-02-19 06:35:27,225 INFO] f1: 0.0729
model saved: [./saved_models/fixmatch/latest_model.pth](https://vscode-remote+ssh-002dremote-002bzy.vscode-resource.vscode-cdn.net/home/yxyuan/thesis/AutoMTL/saved_models/fixmatch/latest_model.pth)
Epoch: 3
[2023-02-19 06:37:10,505 INFO] confusion matrix
[2023-02-19 06:37:10,505 INFO] [[0.00804486 0.34982935 0.00901999 0.15845929 0.00877621 0.02303754
  0.07996099 0.17710873 0.0591175  0.12664554]
 [0.03830152 0.38705335 0.01297112 0.05861478 0.02116985 0.00403818
  0.13864415 0.13179148 0.04674498 0.16067058]
 [0.01099035 0.40896324 0.00915863 0.09048724 0.02088167 0.01721822
  0.07644401 0.1681524  0.09415069 0.10355355]
 [0.02491147 0.36451337 0.01025766 0.09842472 0.02955184 0.01502015
  0.09720357 0.15984858 0.0918305  0.10843815]
 [0.012057   0.49409329 0.00815979 0.05821459 0.02569724 0.01266594
  0.05370844 0.14590184 0.10157106 0.08793082]
 [0.02402146 0.42043653 0.00621875 0.09230582 0.02304597 0.01524204
  0.08669674 0.15376174 0.07877088 0.09950006]
 [0.01211157 0.54110595 0.00489356 0.03609004 0.02789332 0.00893076
  0.07303646 0.11365305 0.07621727 0.10606802]
 [0.03086495 0.38684885 0.01012566 0.07173356 0.01317555 0.0176894
  0.07039161 0.18274979 0.12602172 0.09039893]
 [0.00782779 0.30858611 0.01920254 0.12597847 0.0239726  0.01418787
  0.11876223 0.16964286 0.05418297 0.15765656]
 [0.04349945 0.32167662 0.0152309  0.09869623 0.01815523 0.01669307
  0.11758255 0.17667845 0.05397831 0.13780919]]
[2023-02-19 06:37:10,507 INFO] evaluation metric
[2023-02-19 06:37:10,507 INFO] acc: 0.0991
[2023-02-19 06:37:10,508 INFO] precision: 0.0941
[2023-02-19 06:37:10,508 INFO] recall: 0.0991
[2023-02-19 06:37:10,508 INFO] f1: 0.0764
model saved: [./saved_models/fixmatch/latest_model.pth](https://vscode-remote+ssh-002dremote-002bzy.vscode-resource.vscode-cdn.net/home/yxyuan/thesis/AutoMTL/saved_models/fixmatch/latest_model.pth)
Epoch: 4
[2023-02-19 06:38:48,838 INFO] confusion matrix
[2023-02-19 06:38:48,839 INFO] [[0.00694783 0.34678206 0.01316431 0.1582155  0.0088981  0.01974647
  0.07557289 0.19953681 0.05936129 0.11177474]
 [0.0324278  0.38423886 0.01395007 0.06558982 0.02129222 0.00697504
  0.13240333 0.13754283 0.04478708 0.16079295]
 [0.0097692  0.39040176 0.00708267 0.09268531 0.02686531 0.01734033
  0.07729882 0.17755526 0.09524973 0.10575162]
 [0.0211259  0.36781048 0.01123458 0.10636219 0.03174991 0.01697399
  0.09085358 0.15410917 0.09354011 0.10624008]
 [0.0130313  0.49092681 0.00815979 0.06150286 0.02581902 0.01071733
  0.05468274 0.15150408 0.09292413 0.09073194]
 [0.02389952 0.4229972  0.00414584 0.08596513 0.02292403 0.01719303
  0.07572247 0.1630289  0.08011218 0.10401171]
 [0.01321263 0.53792513 0.00587228 0.03682408 0.02361145 0.00990947
  0.06667482 0.12796672 0.07095669 0.10704673]
 [0.02598512 0.41319995 0.01110162 0.06294986 0.01329755 0.0118336
  0.06502379 0.18933756 0.10479444 0.10247652]
 [0.00684932 0.31152153 0.01700098 0.11925147 0.02005871 0.01394325
  0.11460372 0.20205479 0.04880137 0.14591487]
 [0.04179359 0.33727306 0.01913001 0.10746923 0.02010479 0.01376873
  0.10759108 0.18228342 0.0480078  0.12257829]]
[2023-02-19 06:38:48,841 INFO] evaluation metric
[2023-02-19 06:38:48,841 INFO] acc: 0.0974
[2023-02-19 06:38:48,841 INFO] precision: 0.0921
[2023-02-19 06:38:48,842 INFO] recall: 0.0975
[2023-02-19 06:38:48,842 INFO] f1: 0.0745
model saved: [./saved_models/fixmatch/latest_model.pth](https://vscode-remote+ssh-002dremote-002bzy.vscode-resource.vscode-cdn.net/home/yxyuan/thesis/AutoMTL/saved_models/fixmatch/latest_model.pth)
Epoch: 5
Hhhhhhao commented 1 year ago

You should increase the number of epochs and adjust the learning rate accordingly to get good performance. We set the epochs 5 to easier displaying and debugging.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] commented 1 year ago

This issue was closed because it has been stalled for 5 days with no activity.