search

arthurdouillard / CVPR2021_PLOP

Official code of CVPR 2021's PLOP: Learning without Forgetting for Continual Semantic Segmentation
https://arxiv.org/abs/2011.11390
MIT License
145 stars 23 forks source link

Asking for the data of ilt, mib and plop. #12

Closed schuy1er closed 3 years ago

schuy1er commented 3 years ago

Hello, Hope it'll not disturb you but I'm wondering if you can offer me the data(mIoU) of each step about ILT, MiB and PLOP on 15-5s as well as cityscapes? It's wonderful to follow your impressive work, but I still have to do some analysis over these data.

arthurdouillard commented 3 years ago

Complete results with the format for each line:

step_id,mIoU-background,mIoU-class1,mIoU-class2,...,x (classes not yet seen),x,x,...,overall mIoU

For VOC 15-5s (15-1):

PLOP:

0,0.9440279,0.9104441,0.41364208,0.898608,0.71929836,0.8215646,0.9363354,0.898854,0.93032265,0.45576826,0.84867966,0.58116364,0.89876455,0.8657829,0.86656964,0.8648017,x,x,x,x,x,0.8034142256
1,0.9137345,0.8230578,0.3865131,0.8355453,0.6962276,0.78011644,0.9363182,0.89044285,0.8989261,0.4699988,0.8189739,0.6117257,0.8620734,0.8152025,0.8382983,0.84796643,0.20388034,x,x,x,x,0.7428824306
2,0.91630095,0.83182377,0.36733705,0.82886755,0.6050252,0.71446735,0.9301044,0.88638526,0.91158104,0.41925108,0.690354,0.592483,0.8311134,0.84202087,0.8174002,0.8173679,0.15230848,0.39874703,x,x,x,0.6973855495
3,0.842339,0.7179937,0.31061265,0.7886318,0.62563163,0.67622644,0.9274522,0.8771076,0.8841727,0.4198285,0.7298316,0.57347894,0.8607871,0.8096045,0.78250414,0.8144463,0.09295991,0.52338713,0.16697048,x,x,0.6538929939
4,0.84551114,0.84804404,0.33756274,0.697231,0.6196169,0.63752943,0.8020515,0.8302979,0.8189282,0.2770181,0.6678214,0.5428652,0.7951796,0.7370438,0.7728697,0.7378281,0.10945321,0.36362392,0.122427225,0.20510904,x,0.5884005427
5,0.80289865,0.7747581,0.29458275,0.6158564,0.58670324,0.640667,0.7942619,0.7794818,0.8008924,0.3037926,0.62809366,0.537837,0.76482993,0.67003906,0.7211088,0.70320225,0.14577247,0.45536798,0.11020233,0.2400926,0.10426089,0.5464143753

MiB:

0,0.9433432,0.88860404,0.42194805,0.8960234,0.66153026,0.8292734,0.94626534,0.9017665,0.922483,0.45490792,0.8473392,0.61779964,0.86346644,0.86124754,0.87864864,0.86378247,x,x,x,x,x,0.7999018430709839
1,0.9028324,0.7293984,0.41575187,0.8343833,0.624853,0.7816159,0.902279,0.84762275,0.9134791,0.4341344,0.80821854,0.54598194,0.6596045,0.8350077,0.83645964,0.8356472,0.22356154,x,x,x,x,0.7135782837867737
2,0.89492726,0.43855703,0.39609388,0.6498761,0.47251496,0.6729635,0.7122778,0.6866167,0.854003,0.117564686,0.44825476,0.4242386,0.643971,0.6902108,0.775355,0.8107349,0.14165694,0.26580045,x,x,x,0.5608676075935364
3,0.8413081,0.38468662,0.37346384,0.63944554,0.2793162,0.6691821,0.75155926,0.657113,0.84389746,0.13144453,0.4452766,0.44533768,0.6956522,0.64258647,0.66387403,0.833852,0.14305067,0.25212783,0.17571166,x,x,0.519415020942688
4,0.85543007,0.095600896,0.32580942,0.42438054,0.2646501,0.548344,0.30485547,0.587476,0.67676365,0.06551102,0.4575198,0.3050225,0.24497585,0.44967544,0.31377655,0.81582415,0.049386054,0.17045003,0.1557167,0.18251655,x,0.3646842837333679
5,0.8279441,0.008854385,0.28076664,0.39780357,0.06586075,0.39179677,0.054508347,0.14749165,0.72508705,0.07645087,0.3745203,0.32600558,0.32158142,0.42807162,0.23533955,0.8139699,0.015058636,0.18094094,0.15394227,0.14298517,0.18237005,0.29292139410972595

ILT:

0,0.9440286,0.9104608,0.41364482,0.8986161,0.71928346,0.82157075,0.9363317,0.89885086,0.930327,0.45580387,0.8487623,0.5811669,0.89879215,0.8658049,0.8665504,0.8647938,x,x,x,x,x,0.8034242391586304
1,0.9215822,0.90580595,0.42093217,0.7409449,0.6377902,0.66235733,0.9322979,0.86435443,0.8906912,0.40378168,0.8167565,0.42599306,0.8947986,0.82988644,0.8045869,0.845413,0.20421717,x,x,x,x,0.717775821685791
2,0.9128361,0.7522304,0.20495245,0.013871588,0.63935745,0.60579085,0.90423685,0.6659718,0.01709534,0.19236104,0.000976954,0.47727838,0.02047424,0.43888488,0.3681319,0.72936827,0.23215432,0.078033194,x,x,x,0.4030003249645233
3,0.83098286,0.61585253,0.08945245,0.25996256,0.40457654,0.47056425,0.8307234,0.018446939,0.0005061923,0.085779004,0.1302678,0.44872764,0.00457277,0.522315,0.14488836,0.6241926,0.0001420353,0.09059413,0.1339223,x,x,0.3003405034542084
4,0.83267134,0.0,0.0,0.17624512,0.0,0.020734558,0.0,0.0,0.00031551818,0.0059750257,0.000455689,0.35724148,0.00019222382,0.0054555223,0.0,0.28856176,0.0,0.000182839,0.22132546,0.076872416,x,0.0993114560842514
5,0.7922977,0.020129928,0.0,0.08372341,0.013908777,0.009340274,0.0,0.0,0.0,0.001115043,0.0037631695,0.27111015,0.0,0.1152172,0.0,0.08864881,0.0,0.0,0.21770608,0.09445807,0.087131694,0.08564525842666626

Cityscapes-domain 1-1

About Cityscapes-domain, I have trouble reproducing the exact (or better) results in the paper, but they are still a bit above MiB and even more ILT (see https://github.com/arthurdouillard/CVPR2021_PLOP/issues/9#issuecomment-846571788).

PLOP:

 0,0.9512179,0.49898145,0.8070292,0.0,0.0,1.1055343e-05,0.0,0.0,0.8039116,0.0026853262,0.84018236,0.33359814,0.0,0.73883486,X,0.0,X,0.0,0.0,0.29273247718811035
1,0.95419914,0.5249252,0.7853988,0.0,0.0,0.0002856461,0.0,0.0,0.8246656,0.24420391,0.8456582,0.2893531,0.0,0.75286084,0.0,0.10098645,0.0,0.0,0.0,0.2801334857940674
2,0.94359225,0.48071897,0.772347,0.0,0.0,0.00064745586,0.0,0.0,0.80495065,0.46300852,0.8440916,0.27528957,0.0,0.6996798,0.0,0.08841864,0.0,0.0,0.0,0.28277602791786194
3,0.95042634,0.54959667,0.7900442,0.0,0.0,0.00021482115,0.0,0.0,0.7990772,0.45033205,0.85793465,0.27883098,0.0,0.71048695,0.0,0.036816105,0.0,0.0,0.0,0.28546103835105896
4,0.9513296,0.5186666,0.777814,0.0,0.0,0.0012057272,0.0,0.0,0.8030011,0.40828913,0.86843646,0.30091506,0.0,0.71709764,0.0,0.016383594,0.0,0.0,0.004898417,0.282528281211853
5,0.9531688,0.51628345,0.79524046,0.0,0.0,0.0011286316,0.0,0.0,0.8020942,0.45837492,0.8711227,0.31808183,0.0,0.71495014,0.0,0.011668282,0.0,0.0,0.040963165,0.288582980632782
6,0.9541406,0.5591732,0.80475605,0.0,0.07682736,0.0006287044,0.0,0.0067549828,0.80758935,0.4396642,0.87175375,0.3758609,0.0,0.7312207,0.0,0.17241763,0.0,0.0,0.10407121,0.3107820451259613
7,0.9551862,0.55171406,0.794692,0.0008026284,0.06708697,0.0002482595,0.0,0.002899591,0.8189455,0.38911486,0.8672651,0.41688946,0.0,0.7459022,0.0,0.14566818,0.0,0.0,0.21653163,0.3143656253814697
8,0.9556289,0.56062806,0.81857014,0.009893791,0.20506838,3.9796447e-05,0.0,0.0020180703,0.8279553,0.4524405,0.87851244,0.41741318,0.0,0.75790226,0.0,0.26413968,0.0,0.0,0.216202,0.3350743353366852
9,0.95286685,0.5374765,0.8192359,0.028342495,0.08887138,0.0029712168,0.0,0.035850808,0.8348514,0.50272864,0.87406325,0.40331715,0.0,0.77524775,0.0,0.011796257,0.0,0.0,0.26623574,0.32283446192741394
10,0.95500565,0.5477209,0.8202163,0.01412706,0.21460645,0.012609714,0.0,0.03753798,0.83687043,0.4852281,0.8815881,0.43993372,0.0,0.7670471,0.0,0.31442598,0.0,0.0,0.26341772,0.34685972332954407
11,0.9577117,0.5377769,0.8076013,0.020573705,0.18592055,0.013626974,0.0,0.0644945,0.84033334,0.52973515,0.87054473,0.411278,0.0,0.77666485,0.0,0.13391945,0.00014081779,0.0,0.38666114,0.3440517485141754
12,0.9570291,0.53992057,0.8205026,0.019569602,0.22392574,0.010772013,0.0,0.14813904,0.84343004,0.49129796,0.88374174,0.44682136,0.0,0.77886105,0.0,0.3163202,0.0,0.0,0.38647735,0.3614109456539154
13,0.95934093,0.5701748,0.82185066,0.12874562,0.23590279,0.0026149026,0.0,0.10297459,0.8492526,0.5027345,0.88493484,0.4712517,0.0,0.776796,0.0,0.217919,0.0052429615,0.0,0.39293346,0.3643510341644287
14,0.9599335,0.5709317,0.81944007,0.19978972,0.26682323,0.00447629,0.0,0.093297884,0.8484896,0.4908857,0.8834238,0.47621995,0.0,0.7850075,0.0,0.2880099,0.006964463,0.0,0.4120436,0.373986154794693
15,0.9603913,0.5678387,0.82634544,0.22233616,0.26334554,0.01155901,0.0,0.14075035,0.8483718,0.47693443,0.89063615,0.50630593,0.0,0.7903441,0.02300585,0.3583547,0.052856307,0.0,0.3820531,0.38533836603164673
16,0.95828056,0.56282943,0.8191539,0.20003523,0.2310593,0.009203077,0.0,0.17983277,0.8466069,0.47048312,0.88988256,0.49410498,0.0064627333,0.7799858,0.22048086,0.34843698,0.00043207643,0.0,0.37529787,0.38908252120018005
17,0.95948166,0.5734325,0.82576257,0.24712591,0.24527352,0.011913195,0.0,0.19107169,0.8472153,0.485794,0.89360243,0.51640475,0.012956188,0.79166085,0.08658582,0.39365333,0.5517193,0.0,0.38119,0.4218337833881378
18,0.96101815,0.5799411,0.8341383,0.19611411,0.2994994,0.031680897,0.0,0.22035773,0.8465629,0.470573,0.89465606,0.53046125,0.020974189,0.7880584,0.1771895,0.4612345,0.5966816,0.0,0.3762088,0.43607109785079956
19,0.9556643,0.5238907,0.8205979,0.18254748,0.18493263,0.021399654,0.0,0.18413055,0.845909,0.45854786,0.87567335,0.49903086,0.016382562,0.78576314,0.24384353,0.3584323,0.5213165,0.0,0.38613814,0.4139052927494049
20,0.96167994,0.5761349,0.830038,0.2602308,0.22959839,0.02587215,0.0,0.21224555,0.8512933,0.45518523,0.89416254,0.50962406,0.03731056,0.79244435,0.18459295,0.4264952,0.5577296,0.0,0.3941473,0.4315149784088135

MiB

0,0.9511943,0.50070065,0.8079486,0.0,0.0,0.0,0.0,0.0,0.80452526,0.001340932,0.84039366,0.33528423,0.0,0.7396006,X,0.0,X,0.0,0.0,0.2929992973804474
1,0.9533814,0.518882,0.78350765,0.0,0.0,0.0002976132,0.0,0.0,0.82439333,0.22153744,0.848541,0.29136324,0.0,0.7524534,0.0,0.054789785,0.0,0.0,0.0,0.27627089619636536
2,0.94487596,0.49430624,0.77242565,0.0,0.0,0.00041073063,0.0,0.0,0.8028703,0.4689886,0.84695244,0.27382082,0.0,0.6911339,0.0,0.05326322,0.0,0.0,0.0,0.2815288305282593
3,0.9506075,0.5488296,0.7876915,0.0,0.0,0.00016944438,0.0,0.0,0.7981471,0.45802805,0.8574458,0.26934075,0.0,0.70558596,0.0,0.026629014,0.0,0.0,0.0,0.2843407690525055
4,0.94993263,0.5120624,0.7749934,0.0,0.0,0.0006713311,0.0,0.0,0.8020382,0.42142725,0.8674694,0.29286838,0.0,0.70521796,0.0,0.004139781,0.0,0.0,0.001379524,0.2806421220302582
5,0.95197296,0.51113105,0.7921617,0.0,0.0,0.00070592255,0.0,0.0,0.801085,0.46723092,0.8684399,0.31917998,0.0,0.7060352,0.0,0.0014979454,0.0,0.0,0.018999657,0.28623369336128235
6,0.95268035,0.54761577,0.8018122,0.0,0.039865922,0.0003671433,0.0,0.0053844433,0.80719864,0.4378054,0.87009704,0.37212694,0.0,0.72194886,0.0,0.10220631,0.0,0.0,0.07379772,0.3017319440841675
7,0.95439684,0.54420304,0.7975085,1.7645087e-05,0.036406282,0.00012675382,0.0,0.0015600034,0.81723046,0.3970405,0.8699184,0.4138653,0.0,0.7369248,0.0,0.086554244,0.0,0.0,0.1897097,0.307655930519104
8,0.9547337,0.5553133,0.81639713,0.00554042,0.16992354,5.135856e-06,0.0,0.0010521639,0.82612306,0.4536111,0.87558705,0.41672537,0.0,0.7500283,0.0,0.23335297,0.0,0.0,0.1922294,0.32898011803627014
9,0.95091987,0.5225522,0.8185486,0.014842575,0.078467645,0.002047421,0.0,0.026636895,0.83298403,0.50201845,0.87771636,0.40833527,0.0,0.7714839,0.0,0.00026755576,0.0,0.0,0.2371929,0.3181059956550598
10,0.95325077,0.530708,0.8182606,0.0051759793,0.18738015,0.0087655205,0.0,0.029213749,0.8360167,0.48138264,0.8796331,0.43618593,0.0,0.7612522,0.0,0.22819084,0.0,0.0,0.2216545,0.33563530445098877
11,0.956835,0.53058094,0.8069989,0.014730077,0.17194362,0.011859042,0.0,0.048338376,0.8402284,0.53049564,0.8738097,0.41110477,0.0,0.7722812,0.0,0.10269138,0.0,0.0,0.36311767,0.338685005903244
12,0.9557184,0.52805126,0.8161765,0.010773214,0.20072904,0.011939583,0.0,0.12207835,0.8422609,0.49856365,0.8830183,0.44378492,0.0,0.77452695,0.0,0.33194298,0.0,0.0,0.37575817,0.3576485216617584
13,0.95789176,0.55515826,0.8181979,0.10058745,0.21004426,0.0026534118,0.0,0.08385989,0.846639,0.4969597,0.8834084,0.47024006,0.0,0.7734482,0.0,0.23357351,0.0,0.0,0.39153966,0.35916849970817566
14,0.95814973,0.5528354,0.8162529,0.17028467,0.24935472,0.004500157,0.0,0.08179708,0.84603935,0.48722222,0.8805912,0.4702903,0.0,0.77943736,0.0,0.29253733,0.0,0.0,0.4016419,0.3679439127445221
15,0.95943326,0.5564603,0.8237674,0.18537755,0.24975275,0.010034283,0.0,0.120553896,0.84645873,0.4792964,0.8896918,0.50464827,0.0,0.78393126,0.0032405339,0.33556917,9.347432e-05,0.0,0.38033003,0.37519150972366333
16,0.95723414,0.54933083,0.81835496,0.17325254,0.22428197,0.0086952355,0.0,0.15592073,0.84502965,0.46978694,0.88769007,0.4967775,0.00029280843,0.7766523,0.1489092,0.34084547,0.0,0.0,0.37441304,0.38039299845695496
17,0.95856833,0.56258994,0.82271945,0.2328155,0.22761698,0.009539923,0.0,0.16815734,0.84592366,0.47954232,0.8929125,0.51248693,0.0014795818,0.78812885,0.03704085,0.38035962,0.48157284,0.0,0.37832397,0.4094620645046234
18,0.96031,0.5725947,0.8299889,0.18275313,0.2771108,0.025214985,0.0,0.19686446,0.84476507,0.46969667,0.8925131,0.5258186,0.005375337,0.78566766,0.13183384,0.4301402,0.57021135,0.0,0.37141597,0.4248565435409546
19,0.9553174,0.5260285,0.8163063,0.18641035,0.16102754,0.015468332,0.0,0.16279395,0.84457076,0.45620212,0.8740638,0.48314747,0.0017818625,0.78414404,0.19259822,0.32982394,0.4415426,0.0,0.38632026,0.40092355012893677
20,0.9604027,0.5641127,0.82776076,0.24123996,0.20822123,0.021884276,0.0,0.19355243,0.84999967,0.45944864,0.89303166,0.50607777,0.011034083,0.788951,0.15466943,0.39278364,0.5247078,0.0,0.39576608,0.42071810364723206

ILT

0,0.9513925,0.49971446,0.8068676,0.0,0.0,5.5276105e-05,0.0,0.0,0.8041992,0.00232342,0.8387104,0.3337676,0.0,0.73961,X,0.0,X,0.0,0.0,0.2927435040473938
1,0.94840926,0.48329824,0.7730519,0.0,0.0,3.1464475e-05,0.0,0.0,0.8170426,0.11001786,0.83937365,0.28573895,0.0,0.737318,0.0,0.10074598,0.0,0.0,0.0,0.2681593596935272
2,0.94330084,0.46054697,0.76322275,0.0,0.0,4.699593e-05,0.0,0.0,0.7981394,0.416588,0.8427788,0.24902678,0.0,0.69271076,0.0,0.028455112,0.0,0.0,0.0,0.2734113931655884
3,0.94614303,0.5023591,0.7732386,0.0,0.0,9.85944e-05,0.0,0.0,0.7881512,0.4179446,0.8473946,0.22957039,0.0,0.6936051,0.0,0.0132561615,0.0,0.0,0.0,0.2743031978607178
4,0.9439207,0.4666402,0.74859995,0.0,0.0,0.00017914167,0.0,0.0,0.78772676,0.40062088,0.8428611,0.25206685,0.0,0.6923873,0.0,0.017206687,0.0,0.0,0.0,0.27116891741752625
5,0.9433114,0.46280047,0.7675174,0.0,0.0,7.89331e-05,0.0,0.0,0.7820021,0.3712087,0.8465283,0.27679455,0.0,0.69185907,0.0,0.0102022225,0.0,0.0,0.0,0.2711738646030426
6,0.9425915,0.47947177,0.77265596,0.0,0.0,0.0001320567,0.0,0.0,0.78673506,0.347506,0.835928,0.31159672,0.0,0.7009333,0.0,0.0,0.0,0.0,0.0008757047,0.27254873514175415
7,0.9456175,0.4827616,0.7753969,0.0,0.0,2.7026042e-06,0.0,0.0,0.79665244,0.3364162,0.83691794,0.34671035,0.0,0.7115451,0.0,0.0,0.0,0.0,0.0046473304,0.2756141126155853
8,0.9440899,0.48022997,0.78796226,0.0,0.0,1.2840778e-06,0.0,0.0,0.7994624,0.36648473,0.83066654,0.35446396,0.0,0.7129533,0.0,4.9144535e-05,0.0,0.0,0.0035799895,0.27789175510406494
9,0.9417322,0.46623823,0.7925627,0.0,0.0,4.8882903e-06,0.0,4.984374e-05,0.81043226,0.40275374,0.8459737,0.3577139,0.0,0.73859745,0.0,0.0,0.0,0.0,0.0080980975,0.28232407569885254
10,0.9430355,0.4548832,0.7874148,0.0,0.0003799268,0.00016425406,0.0,0.0,0.814454,0.41264513,0.8449078,0.38105935,0.0,0.73000073,0.0,0.0,0.0,0.0,0.0035281465,0.28276172280311584
11,0.945887,0.45989555,0.7755233,0.0,0.000113521135,0.0012801331,0.0,0.0036960228,0.81321716,0.4271867,0.842766,0.3379505,0.0,0.73511016,0.0,0.0,0.0,0.0,0.030654948,0.2828042507171631
12,0.9430172,0.43678486,0.7733995,0.0,0.0,0.002051303,0.0,0.010135501,0.8132779,0.4181528,0.84903765,0.37456578,0.0,0.7314073,0.0,0.016815655,0.0,0.0,0.05499062,0.28545451164245605
13,0.9437569,0.4563224,0.78021014,0.00019759884,0.0017164124,0.0017275938,0.0,0.007844851,0.8137071,0.43018854,0.84601015,0.3919615,0.0,0.72393304,0.0,0.10189507,0.0,0.0,0.07288214,0.2932817339897156
14,0.9441838,0.4590645,0.7796174,1.796812e-05,0.005119001,0.0016771237,0.0,0.014238953,0.81418407,0.431642,0.8515754,0.3995093,0.0,0.7228796,0.0,0.056441758,0.0,0.0,0.08835329,0.293079137802124
15,0.9448985,0.45020282,0.785752,0.0009844842,0.007907734,0.0021744352,0.0,0.014928838,0.8147242,0.41792622,0.8567481,0.41458634,0.0,0.73036057,0.0,0.036414813,0.0,0.0,0.1004708,0.2935831546783447
16,0.94465554,0.46628392,0.78556925,0.000983851,0.008386092,0.0024783288,0.0,0.027130088,0.8134208,0.41187975,0.85462755,0.4035749,0.0,0.71878463,0.0,0.0919799,0.0,0.0,0.1140631,0.2970430552959442
17,0.9453012,0.46544704,0.78402394,0.0036091786,0.011302207,0.00300923,0.0,0.023852572,0.81419414,0.40027153,0.86165285,0.4071051,0.0,0.72746116,0.0,0.088316634,0.0,0.0,0.12610194,0.29798153042793274
18,0.94754493,0.4806536,0.7874799,0.007446311,0.016414145,0.0047390447,0.0,0.029331857,0.8160001,0.41524193,0.8588218,0.42143592,0.0,0.7254876,0.0,0.06029996,0.0012842048,0.0,0.16304503,0.3018540143966675
19,0.9457779,0.47231638,0.78608143,0.007718707,0.0075703454,0.0057262746,0.0,0.0301497,0.81426793,0.3930644,0.8537473,0.415241,0.0,0.7284761,0.0,0.06518377,0.0,0.0,0.1590552,0.29917773604393005
20,0.9472265,0.46986875,0.7873244,0.0033675174,0.006748041,0.004218687,0.0,0.028913114,0.8186388,0.39995557,0.85547185,0.40221864,0.0,0.7284133,0.0,0.09008952,0.0,0.0,0.19065706,0.3017427623271942