search

med-air / Endo-FM

[MICCAI'23] Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train
Apache License 2.0
146 stars 14 forks source link

Question about pretrain: #16

Closed Starleil closed 2 months ago

Starleil commented 2 months ago

Thanks for your great work. I have some questions about the pre-training phase.

  1. How can I be sure that the pre-training is well done? Is the loss function no longer decreasing?

  2. Could you provide a log of the pre-training, e.g. log.txt?

Hope to hear from you!

Kyfafyd commented 2 months ago

Hi @Starleil , thanks for your interest!

  1. In our experiments, we find that the pre-training is good enough with 30 pre-training epochs. You may obtain slightly better pre-training model with longer pre-training schedule, but cost much more time.
  2. Unfortunately, the log of pre-training is lost currently. You may attach your training log here and I can help you confirm if it works properly.
Starleil commented 2 months ago

Thank you for your reply.

Instead of using ViT and your datasets, I used my own model and dataset.175 pre-training epochs were used. And other hyperparameter settings are consistent with that in your work. Here's my pre-training log, I'm still trying to fine-tune on the weights obtained from the pre-training phase for the downstream task.

It looks like the losses can still be reduced in pre-training phase.

{"train_loss": 11.145007224846387, "train_lr": 1.1677210515951161e-06, "train_wd": 0.04000327044741349, "epoch": 0}
{"train_loss": 11.037506526536516, "train_lr": 3.512394151240646e-06, "train_wd": 0.04002297026083254, "epoch": 1}
{"train_loss": 10.884685891819752, "train_lr": 5.857067250886177e-06, "train_wd": 0.04006240658371926, "epoch": 2}
{"train_loss": 10.593821073141624, "train_lr": 8.201740350531706e-06, "train_wd": 0.04012157509143649, "epoch": 3}
{"train_loss": 10.269831515985524, "train_lr": 1.0546413450177236e-05, "train_wd": 0.040200469295490565, "epoch": 4}
{"train_loss": 10.457059646215965, "train_lr": 1.2891086549822765e-05, "train_wd": 0.040299080544242936, "epoch": 5}
{"train_loss": 10.424536827668117, "train_lr": 1.5235759649468296e-05, "train_wd": 0.040417398023858936, "epoch": 6}
{"train_loss": 10.449955673668327, "train_lr": 1.758043274911383e-05, "train_wd": 0.040555408759493546, "epoch": 7}
{"train_loss": 10.49285593308176, "train_lr": 1.9925105848759353e-05, "train_wd": 0.04071309761671448, "epoch": 8}
{"train_loss": 10.474190158793933, "train_lr": 2.2269778948404883e-05, "train_wd": 0.0408904473031615, "epoch": 9}
{"train_loss": 10.402376867028984, "train_lr": 2.3437281864935397e-05, "train_wd": 0.04108743837044288, "epoch": 10}
{"train_loss": 10.2724345287313, "train_lr": 2.343596791215398e-05, "train_wd": 0.041304049216268277, "epoch": 11}
{"train_loss": 10.011705452375837, "train_lr": 2.3433337569115934e-05, "train_wd": 0.04154025608681742, "epoch": 12}
{"train_loss": 9.625769269747996, "train_lr": 2.3429391144503684e-05, "train_wd": 0.04179603307934518, "epoch": 13}
{"train_loss": 9.17042028246902, "train_lr": 2.3424129101447734e-05, "train_wd": 0.042071352145022015, "epoch": 14}
{"train_loss": 8.99858932858064, "train_lr": 2.3417552057472073e-05, "train_wd": 0.04236618309200972, "epoch": 15}
{"train_loss": 8.897212446518145, "train_lr": 2.340966078442201e-05, "train_wd": 0.04268049358877253, "epoch": 16}
{"train_loss": 8.793417611147163, "train_lr": 2.3400456208373473e-05, "train_wd": 0.043014249167622594, "epoch": 17}
{"train_loss": 8.380262612670737, "train_lr": 2.3389939409524178e-05, "train_wd": 0.043367413228499574, "epoch": 18}
{"train_loss": 7.941754791054513, "train_lr": 2.33781116220672e-05, "train_wd": 0.04373994704298433, "epoch": 19}
{"train_loss": 7.562608569938679, "train_lr": 2.3364974234045754e-05, "train_wd": 0.044131809758546074, "epoch": 20}
{"train_loss": 7.181109926675561, "train_lr": 2.335052878719061e-05, "train_wd": 0.04454295840302225, "epoch": 21}
{"train_loss": 6.763061369497945, "train_lr": 2.3334776976739e-05, "train_wd": 0.04497334788933061, "epoch": 22}
{"train_loss": 6.303844705184926, "train_lr": 2.331772065123572e-05, "train_wd": 0.04542293102041395, "epoch": 23}
{"train_loss": 5.55276427650702, "train_lr": 2.3299361812316207e-05, "train_wd": 0.04589165849441581, "epoch": 24}
{"train_loss": 5.18355380956895, "train_lr": 2.327970261447158e-05, "train_wd": 0.04637947891008632, "epoch": 25}
{"train_loss": 5.195965325503837, "train_lr": 2.3258745364795902e-05, "train_wd": 0.046886338772419794, "epoch": 26}
{"train_loss": 5.255149855623095, "train_lr": 2.323649252271529e-05, "train_wd": 0.04741218249852052, "epoch": 27}
{"train_loss": 5.0443541683119735, "train_lr": 2.3212946699699486e-05, "train_wd": 0.047956952423698275, "epoch": 28}
{"train_loss": 4.784220147829044, "train_lr": 2.3188110658955176e-05, "train_wd": 0.048520588807791636, "epoch": 29}
{"train_loss": 4.492404041014944, "train_lr": 2.3161987315101898e-05, "train_wd": 0.04910302984171955, "epoch": 30}
{"train_loss": 4.3198837515778274, "train_lr": 2.3134579733829826e-05, "train_wd": 0.04970421165425913, "epoch": 31}
{"train_loss": 4.251832075632151, "train_lr": 2.3105891131540202e-05, "train_wd": 0.05032406831904994, "epoch": 32}
{"train_loss": 4.168014338098251, "train_lr": 2.307592487496764e-05, "train_wd": 0.050962531861823555, "epoch": 33}
{"train_loss": 4.145870522740163, "train_lr": 2.304468448078525e-05, "train_wd": 0.05161953226785763, "epoch": 34}
{"train_loss": 4.185582927587073, "train_lr": 2.3012173615191836e-05, "train_wd": 0.052294997489653924, "epoch": 35}
{"train_loss": 4.1816324561081455, "train_lr": 2.2978396093481537e-05, "train_wd": 0.05298885345483907, "epoch": 36}
{"train_loss": 4.253266592170271, "train_lr": 2.294335587959633e-05, "train_wd": 0.05370102407428749, "epoch": 37}
{"train_loss": 4.2838687980131995, "train_lr": 2.2907057085660715e-05, "train_wd": 0.05443143125046523, "epoch": 38}
{"train_loss": 4.2500541245960815, "train_lr": 2.2869503971499025e-05, "train_wd": 0.05517999488599459, "epoch": 39}
{"train_loss": 4.2146597986165775, "train_lr": 2.283070094413577e-05, "train_wd": 0.05594663289243729, "epoch": 40}
{"train_loss": 4.169897076143368, "train_lr": 2.2790652557278203e-05, "train_wd": 0.056731261199296765, "epoch": 41}
{"train_loss": 4.1490594382322055, "train_lr": 2.2749363510782143e-05, "train_wd": 0.05753379376323726, "epoch": 42}
{"train_loss": 4.159353471373323, "train_lr": 2.2706838650100214e-05, "train_wd": 0.05835414257751926, "epoch": 43}
{"train_loss": 4.116434381585422, "train_lr": 2.2663082965713404e-05, "train_wd": 0.059192217681650915, "epoch": 44}
{"train_loss": 4.090419226866264, "train_lr": 2.261810159254523e-05, "train_wd": 0.060047927171252825, "epoch": 45}
{"train_loss": 4.083452826018721, "train_lr": 2.257189980935935e-05, "train_wd": 0.060921177208136404, "epoch": 46}
{"train_loss": 4.064707929204925, "train_lr": 2.2524483038139777e-05, "train_wd": 0.06181187203059446, "epoch": 47}
{"train_loss": 4.067022484802981, "train_lr": 2.2475856843454886e-05, "train_wd": 0.06271991396390254, "epoch": 48}
{"train_loss": 4.041212570257094, "train_lr": 2.242602693180425e-05, "train_wd": 0.06364520343102997, "epoch": 49}
{"train_loss": 3.9895897841305907, "train_lr": 2.23749991509489e-05, "train_wd": 0.06458763896355955, "epoch": 50}
{"train_loss": 3.989341023622516, "train_lr": 2.2322779489225185e-05, "train_wd": 0.06554711721281499, "epoch": 51}
{"train_loss": 3.957296126399903, "train_lr": 2.2269374074841916e-05, "train_wd": 0.06652353296119377, "epoch": 52}
{"train_loss": 3.9233194697780287, "train_lr": 2.2214789175161293e-05, "train_wd": 0.06751677913370603, "epoch": 53}
{"train_loss": 3.879504623089543, "train_lr": 2.215903119596325e-05, "train_wd": 0.06852674680971588, "epoch": 54}
{"train_loss": 3.8724829544437918, "train_lr": 2.2102106680693938e-05, "train_wd": 0.06955332523488654, "epoch": 55}
{"train_loss": 3.822613398759599, "train_lr": 2.2044022309697558e-05, "train_wd": 0.0705964018333251, "epoch": 56}
{"train_loss": 3.8000583280650337, "train_lr": 2.1984784899432604e-05, "train_wd": 0.07165586221992808, "epoch": 57}
{"train_loss": 3.7846899017461837, "train_lr": 2.1924401401671845e-05, "train_wd": 0.07273159021292523, "epoch": 58}
{"train_loss": 3.7747771618716675, "train_lr": 2.1862878902686383e-05, "train_wd": 0.0738234678466196, "epoch": 59}
{"train_loss": 3.751588183413614, "train_lr": 2.1800224622414352e-05, "train_wd": 0.07493137538432441, "epoch": 60}
{"train_loss": 3.7327470386796815, "train_lr": 2.173644591361327e-05, "train_wd": 0.07605519133149316, "epoch": 61}
{"train_loss": 3.7706971036324037, "train_lr": 2.1671550260997405e-05, "train_wd": 0.07719479244904302, "epoch": 62}
{"train_loss": 3.772595203672279, "train_lr": 2.1605545280359342e-05, "train_wd": 0.07835005376686921, "epoch": 63}
{"train_loss": 3.594449585332997, "train_lr": 2.15384387176762e-05, "train_wd": 0.0795208485975497, "epoch": 64}
{"train_loss": 3.5587036410576403, "train_lr": 2.147023844820067e-05, "train_wd": 0.08070704855023748, "epoch": 65}
{"train_loss": 3.526233136165017, "train_lr": 2.140095247553672e-05, "train_wd": 0.08190852354474074, "epoch": 66}
{"train_loss": 3.5221100848649685, "train_lr": 2.1330588930700455e-05, "train_wd": 0.0831251418257868, "epoch": 67}
{"train_loss": 3.5263954072767807, "train_lr": 2.1259156071165842e-05, "train_wd": 0.08435676997747109, "epoch": 68}
{"train_loss": 3.593159816878068, "train_lr": 2.1186662279895676e-05, "train_wd": 0.08560327293788769, "epoch": 69}
{"train_loss": 3.5832324663771127, "train_lr": 2.1113116064357756e-05, "train_wd": 0.08686451401393999, "epoch": 70}
{"train_loss": 3.434265423960876, "train_lr": 2.1038526055526574e-05, "train_wd": 0.08814035489633137, "epoch": 71}
{"train_loss": 3.402755072163149, "train_lr": 2.0962901006870394e-05, "train_wd": 0.0894306556747314, "epoch": 72}
{"train_loss": 3.3968813672840223, "train_lr": 2.0886249793323947e-05, "train_wd": 0.09073527485311952, "epoch": 73}
{"train_loss": 3.382854469323389, "train_lr": 2.080858141024707e-05, "train_wd": 0.09205406936530103, "epoch": 74}
{"train_loss": 3.4471789240846875, "train_lr": 2.0729904972368823e-05, "train_wd": 0.0933868945905963, "epoch": 75}
{"train_loss": 3.4841385289814, "train_lr": 2.065022971271799e-05, "train_wd": 0.09473360436969955, "epoch": 76}
{"train_loss": 3.454034464801674, "train_lr": 2.0569564981539613e-05, "train_wd": 0.09609405102070774, "epoch": 77}
{"train_loss": 3.4892207341515546, "train_lr": 2.0487920245197538e-05, "train_wd": 0.09746808535531465, "epoch": 78}
{"train_loss": 3.5121645123509597, "train_lr": 2.0405305085063558e-05, "train_wd": 0.09885555669517147, "epoch": 79}
{"train_loss": 3.421392672499666, "train_lr": 2.0321729196393027e-05, "train_wd": 0.10025631288841046, "epoch": 80}
{"train_loss": 3.3127535295640937, "train_lr": 2.023720238718699e-05, "train_wd": 0.10167020032633005, "epoch": 81}
{"train_loss": 3.220065831270294, "train_lr": 2.0151734577041346e-05, "train_wd": 0.10309706396023939, "epoch": 82}
{"train_loss": 3.184298188347379, "train_lr": 2.0065335795982503e-05, "train_wd": 0.10453674731846177, "epoch": 83}
{"train_loss": 3.256678740662618, "train_lr": 1.9978016183290525e-05, "train_wd": 0.10598909252349348, "epoch": 84}
{"train_loss": 3.3131491984668484, "train_lr": 1.9889785986309107e-05, "train_wd": 0.10745394030931606, "epoch": 85}
{"train_loss": 3.2574761250542763, "train_lr": 1.9800655559243063e-05, "train_wd": 0.10893113003886247, "epoch": 86}
{"train_loss": 3.172030986575141, "train_lr": 1.971063536194319e-05, "train_wd": 0.11042049972163258, "epoch": 87}
{"train_loss": 3.163212789040489, "train_lr": 1.9619735958678748e-05, "train_wd": 0.11192188603145695, "epoch": 88}
{"train_loss": 3.1105913374844323, "train_lr": 1.9527968016897785e-05, "train_wd": 0.11343512432440753, "epoch": 89}
{"train_loss": 3.0073506408517696, "train_lr": 1.943534230597511e-05, "train_wd": 0.11496004865685294, "epoch": 90}
{"train_loss": 2.986253882815763, "train_lr": 1.934186969594859e-05, "train_wd": 0.11649649180365593, "epoch": 91}
{"train_loss": 2.987504513150819, "train_lr": 1.9247561156243425e-05, "train_wd": 0.11804428527651101, "epoch": 92}
{"train_loss": 2.948495490635376, "train_lr": 1.9152427754384948e-05, "train_wd": 0.11960325934242222, "epoch": 93}
{"train_loss": 2.913963283307049, "train_lr": 1.9056480654699637e-05, "train_wd": 0.12117324304231475, "epoch": 94}
{"train_loss": 2.8784945857615805, "train_lr": 1.895973111700502e-05, "train_wd": 0.12275406420978403, "epoch": 95}
{"train_loss": 2.863154280521318, "train_lr": 1.8862190495288377e-05, "train_wd": 0.12434554948997481, "epoch": 96}
{"train_loss": 2.744211545937604, "train_lr": 1.876387023637407e-05, "train_wd": 0.12594752435859172, "epoch": 97}
{"train_loss": 2.685381377558285, "train_lr": 1.866478187858044e-05, "train_wd": 0.12755981314103781, "epoch": 98}
{"train_loss": 2.671346365558497, "train_lr": 1.8564937050365636e-05, "train_wd": 0.12918223903167925, "epoch": 99}
{"train_loss": 2.6466586590497667, "train_lr": 1.846434746896289e-05, "train_wd": 0.13081462411323386, "epoch": 100}
{"train_loss": 2.5955901564663555, "train_lr": 1.8363024939005613e-05, "train_wd": 0.13245678937628258, "epoch": 101}
{"train_loss": 2.5735518680286553, "train_lr": 1.8260981351141908e-05, "train_wd": 0.13410855473889824, "epoch": 102}
{"train_loss": 2.637731340718688, "train_lr": 1.815822868063925e-05, "train_wd": 0.13576973906639536, "epoch": 103}
{"train_loss": 2.6152052968188024, "train_lr": 1.8054778985979143e-05, "train_wd": 0.1374401601911929, "epoch": 104}
{"train_loss": 2.6275482740084546, "train_lr": 1.795064440744189e-05, "train_wd": 0.13911963493279006, "epoch": 105}
{"train_loss": 2.6413831488373556, "train_lr": 1.7845837165681995e-05, "train_wd": 0.14080797911785575, "epoch": 106}
{"train_loss": 2.6507019021773215, "train_lr": 1.7740369560293967e-05, "train_wd": 0.14250500760042414, "epoch": 107}
{"train_loss": 2.717691021662186, "train_lr": 1.7634253968368933e-05, "train_wd": 0.14421053428219835, "epoch": 108}
{"train_loss": 2.718808036124006, "train_lr": 1.7527502843042104e-05, "train_wd": 0.14592437213295817, "epoch": 109}
{"train_loss": 2.697541460479192, "train_lr": 1.7420128712031324e-05, "train_wd": 0.14764633321107007, "epoch": 110}
{"train_loss": 2.705559002731748, "train_lr": 1.731214417616696e-05, "train_wd": 0.14937622868409717, "epoch": 111}
{"train_loss": 2.664180720966857, "train_lr": 1.720356190791306e-05, "train_wd": 0.1511138688495065, "epoch": 112}
{"train_loss": 2.6953112960614636, "train_lr": 1.7094394649880236e-05, "train_wd": 0.15285906315547265, "epoch": 113}
{"train_loss": 2.748422152780759, "train_lr": 1.6984655213330232e-05, "train_wd": 0.15461162022177338, "epoch": 114}
{"train_loss": 2.723985052121231, "train_lr": 1.687435647667251e-05, "train_wd": 0.15637134786077686, "epoch": 115}
{"train_loss": 2.6979884345773755, "train_lr": 1.6763511383952872e-05, "train_wd": 0.15813805309851722, "epoch": 116}
{"train_loss": 2.6868778984074595, "train_lr": 1.665213294333433e-05, "train_wd": 0.15991154219585607, "epoch": 117}
{"train_loss": 2.716313867717297, "train_lr": 1.6540234225570794e-05, "train_wd": 0.16169162066972842, "epoch": 118}
{"train_loss": 2.6817888927149767, "train_lr": 1.642782836247292e-05, "train_wd": 0.16347809331447002, "epoch": 119}
{"train_loss": 2.6658148121325853, "train_lr": 1.6314928545367098e-05, "train_wd": 0.16527076422322381, "epoch": 120}
{"train_loss": 2.6416995768542852, "train_lr": 1.620154802354744e-05, "train_wd": 0.167069436809423, "epoch": 121}
{"train_loss": 2.628747707491941, "train_lr": 1.60877001027209e-05, "train_wd": 0.16887391382834924, "epoch": 122}
{"train_loss": 2.6207277974152934, "train_lr": 1.597339814344575e-05, "train_wd": 0.1706839973987626, "epoch": 123}
{"train_loss": 2.5851111001789744, "train_lr": 1.5858655559563722e-05, "train_wd": 0.1724994890246024, "epoch": 124}
{"train_loss": 2.6036041281418263, "train_lr": 1.574348581662575e-05, "train_wd": 0.1743201896167526, "epoch": 125}
{"train_loss": 2.6579781082175242, "train_lr": 1.562790243031183e-05, "train_wd": 0.17614589951487586, "epoch": 126}
{"train_loss": 2.651791556972981, "train_lr": 1.5511918964844773e-05, "train_wd": 0.17797641850930834, "epoch": 127}
{"train_loss": 2.6283841241943087, "train_lr": 1.5395549031398562e-05, "train_wd": 0.17981154586301384, "epoch": 128}
{"train_loss": 2.6747733826499696, "train_lr": 1.5278806286500783e-05, "train_wd": 0.18165108033359806, "epoch": 129}
{"train_loss": 2.6972668114872382, "train_lr": 1.5161704430430199e-05, "train_wd": 0.18349482019537702, "epoch": 130}
{"train_loss": 2.68277711074022, "train_lr": 1.5044257205608805e-05, "train_wd": 0.18534256326149728, "epoch": 131}
{"train_loss": 2.6618278197610503, "train_lr": 1.4926478394989173e-05, "train_wd": 0.18719410690611002, "epoch": 132}
{"train_loss": 2.6378507621310106, "train_lr": 1.4808381820436924e-05, "train_wd": 0.1890492480865889, "epoch": 133}
{"train_loss": 2.621205062983714, "train_lr": 1.468998134110872e-05, "train_wd": 0.19090778336579872, "epoch": 134}
{"train_loss": 2.5798935980372266, "train_lr": 1.4571290851825744e-05, "train_wd": 0.19276950893440195, "epoch": 135}
{"train_loss": 2.550436881323176, "train_lr": 1.445232428144317e-05, "train_wd": 0.19463422063321, "epoch": 136}
{"train_loss": 2.5033199050490627, "train_lr": 1.4333095591215527e-05, "train_wd": 0.1965017139755722, "epoch": 137}
{"train_loss": 2.4575397568312414, "train_lr": 1.4213618773158215e-05, "train_wd": 0.19837178416979842, "epoch": 138}
{"train_loss": 2.435237883332276, "train_lr": 1.409390784840557e-05, "train_wd": 0.20024422614161855, "epoch": 139}
{"train_loss": 2.4230483482907137, "train_lr": 1.3973976865565384e-05, "train_wd": 0.20211883455666937, "epoch": 140}
{"train_loss": 2.401864389912164, "train_lr": 1.3853839899070263e-05, "train_wd": 0.20399540384301343, "epoch": 141}
{"train_loss": 2.4309875333029045, "train_lr": 1.3733511047525865e-05, "train_wd": 0.2058737282136818, "epoch": 142}
{"train_loss": 2.444163305495482, "train_lr": 1.3613004432056423e-05, "train_wd": 0.2077536016892398, "epoch": 143}
{"train_loss": 2.4323832388341504, "train_lr": 1.3492334194647576e-05, "train_wd": 0.2096348181203771, "epoch": 144}
{"train_loss": 2.410690539765839, "train_lr": 1.337151449648669e-05, "train_wd": 0.21151717121051242, "epoch": 145}
{"train_loss": 2.43082092700311, "train_lr": 1.3250559516301057e-05, "train_wd": 0.21340045453841738, "epoch": 146}
{"train_loss": 2.4249084478395737, "train_lr": 1.3129483448693883e-05, "train_wd": 0.2152844615808523, "epoch": 147}
{"train_loss": 2.4328909795315754, "train_lr": 1.3008300502478491e-05, "train_wd": 0.21716898573521431, "epoch": 148}
{"train_loss": 2.4321212281468094, "train_lr": 1.2887024899010981e-05, "train_wd": 0.21905382034219273, "epoch": 149}
{"train_loss": 2.383576485080268, "train_lr": 1.276567087052107e-05, "train_wd": 0.22093875870843338, "epoch": 150}
{"train_loss": 2.3963443780295903, "train_lr": 1.2644252658442075e-05, "train_wd": 0.2228235941292029, "epoch": 151}
{"train_loss": 2.3886648261763175, "train_lr": 1.2522784511739547e-05, "train_wd": 0.22470811991105674, "epoch": 152}
{"train_loss": 2.383002373374011, "train_lr": 1.2401280685239046e-05, "train_wd": 0.22659212939450687, "epoch": 153}
{"train_loss": 2.401393147949325, "train_lr": 1.2279755437953306e-05, "train_wd": 0.2284754159766824, "epoch": 154}
{"train_loss": 2.384430720148183, "train_lr": 1.2158223031408969e-05, "train_wd": 0.23035777313398698, "epoch": 155}
{"train_loss": 2.3663315649799457, "train_lr": 1.2036697727972739e-05, "train_wd": 0.2322389944447456, "epoch": 156}
{"train_loss": 2.3284296937521516, "train_lr": 1.1915193789177829e-05, "train_wd": 0.2341188736118425, "epoch": 157}
{"train_loss": 2.3015948216714395, "train_lr": 1.1793725474050136e-05, "train_wd": 0.2359972044853427, "epoch": 158}
{"train_loss": 2.2722683083706015, "train_lr": 1.1672307037435049e-05, "train_wd": 0.2378737810850989, "epoch": 159}
{"train_loss": 2.242429586689616, "train_lr": 1.1550952728324409e-05, "train_wd": 0.23974839762333974, "epoch": 160}
{"train_loss": 2.2522260477018405, "train_lr": 1.1429676788184452e-05, "train_wd": 0.24162084852723734, "epoch": 161}
{"train_loss": 2.212131650343518, "train_lr": 1.1308493449284493e-05, "train_wd": 0.24349092846144915, "epoch": 162}
{"train_loss": 2.2289931375100682, "train_lr": 1.1187416933026627e-05, "train_wd": 0.24535843235063598, "epoch": 163}
{"train_loss": 2.2177799867302315, "train_lr": 1.1066461448276852e-05, "train_wd": 0.24722315540195233, "epoch": 164}
{"train_loss": 2.2082053950127323, "train_lr": 1.0945641189697629e-05, "train_wd": 0.24908489312750187, "epoch": 165}
{"train_loss": 2.163073701461352, "train_lr": 1.082497033608196e-05, "train_wd": 0.2509434413667627, "epoch": 166}
{"train_loss": 2.1376130190858826, "train_lr": 1.0704463048689566e-05, "train_wd": 0.25279859630897694, "epoch": 167}
{"train_loss": 2.1109347016271935, "train_lr": 1.0584133469584906e-05, "train_wd": 0.2546501545154996, "epoch": 168}
{"train_loss": 2.095803860648293, "train_lr": 1.0463995719977617e-05, "train_wd": 0.25649791294210755, "epoch": 169}
{"train_loss": 2.0839627327011248, "train_lr": 1.0344063898565256e-05, "train_wd": 0.2583416689612673, "epoch": 170}
{"train_loss": 2.0612345722557737, "train_lr": 1.0224352079878848e-05, "train_wd": 0.26018122038435393, "epoch": 171}
{"train_loss": 2.0319688779881266, "train_lr": 1.0104874312631078e-05, "train_wd": 0.26201636548382484, "epoch": 172}
{"train_loss": 2.0379330367008786, "train_lr": 9.985644618067653e-06, "train_wd": 0.2638469030153387, "epoch": 173}
{"train_loss": 1.9891508812478345, "train_lr": 9.8666769883219e-06, "train_wd": 0.26567263223982707, "epoch": 174}
Kyfafyd commented 2 months ago

Genrally speaking, longer pre-training schedule will produce better pre-trained model, as illustrated in Figure 9 in SimCLR (https://arxiv.org/pdf/2002.05709). There is a trade-off between training cost and downstream performance. I would suggest you can try the pre-trained model on your downstream task. The pre-training is done when you obtain satisfactory downstream performance.

Starleil commented 2 months ago

Thanks, I will try the downstream task.