Expert Lip-Sync Discriminator is not converging on LRS2?

[Mayur28]

Hi,

I have experimenting with the code, but regardless of what I try with the expert lip-sync discriminator, the loss does not seem to go below 0.68. I am using the LRS2 (main) dataset. For some reason, even when I train the model exactly as it is, it is not showing signs of convergences. In previous issues, it was said that the model starts to converge after around 150K - 100K steps - I currently have the original model training now training for 350K steps but the loss has not gone below 0.68.

If anyone is experiencing this issue (or has not experienced this issue) when training with LRS2, please let me know, as this will also give an indication if there is an issue on my side.

Any assistance would be highly appreciated. Thanks

[voducman]

What batch size are you using right now? When I use batch size is 32, it was not converged and I increase it up to 64, It works.

[Mayur28]

Hi @voducman! I've tried 64 and 128 several times, both did not work (the original implementation uses 64). Did you use the code from the repo as is working any modifications? After how many steps did your model begin to converge? I've tried the original several times (right now, one of them is at 270K steps, and has still not started to converge)

[voducman]

I am training right now. This is my training logs ( I was modified the original repo):

Loss: 0.7035631441005935 - learning rate: 0.0001 - val loss: 0.6879662836299223 at epoch: 0: 100%|█| Loss: 0.6925660769976 - learning rate: 0.0001 - val loss: 0.69101863748887 at epoch: 1: 100%|█| 717/7 Loss: 0.6937094443845283 - learning rate: 0.0001 - val loss: 0.6934942813480601 at epoch: 2: 100%|█| Loss: 0.6964753221102173 - learning rate: 0.0001 - val loss: 0.6958373785018921 at epoch: 3: 100%|█| Loss: 0.6948814049591745 - learning rate: 0.0001 - val loss: 0.6927358402925379 at epoch: 4: 100%|█| Loss: 0.6957624335695245 - learning rate: 0.0001 - val loss: 0.6927358402925379 at epoch: 5: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000005.pth Loss: 0.6957362132923706 - learning rate: 0.0001 - val loss: 0.6922311712713802 at epoch: 5: 100%|█| Loss: 0.6941021396525235 - learning rate: 0.0001 - val loss: 0.6958037965437945 at epoch: 6: 100%|█| Loss: 0.6915607976946705 - learning rate: 0.0001 - val loss: 0.690644127481124 at epoch: 7: 100%|█| 7 Loss: 0.6906726746047058 - learning rate: 0.0001 - val loss: 0.6915773889597725 at epoch: 8: 100%|█| Loss: 0.690080951946051 - learning rate: 0.0001 - val loss: 0.6910958255038542 at epoch: 9: 100%|█| 7 Loss: 0.6887990220465474 - learning rate: 0.0001 - val loss: 0.6910958255038542 at epoch: 10: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000010.pth Loss: 0.688880747082842 - learning rate: 0.0001 - val loss: 0.6884867303511676 at epoch: 10: 100%|█| Loss: 0.6886253223093317 - learning rate: 0.0001 - val loss: 0.6820760369300842 at epoch: 11: 100%|█| Loss: 0.6794873835983968 - learning rate: 0.0001 - val loss: 0.6745607607504901 at epoch: 12: 100%|█| Loss: 0.6670064611913769 - learning rate: 0.0001 - val loss: 0.676604407675126 at epoch: 13: 100%|█| Loss: 0.6498011181045277 - learning rate: 0.0001 - val loss: 0.6408151738783893 at epoch: 14: 100%|█| Loss: 0.6376869129901491 - learning rate: 0.0001 - val loss: 0.6408151738783893 at epoch: 15: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000015.pth Loss: 0.6374346476718472 - learning rate: 0.0001 - val loss: 0.6302909886135775 at epoch: 15: 100%|█| Loss: 0.6209243929203252 - learning rate: 0.0001 - val loss: 0.5890622138977051 at epoch: 16: 100%|█| Loss: 0.5941242532001878 - learning rate: 0.0001 - val loss: 0.5624779112198773 at epoch: 17: 100%|█| Loss: 0.56167208396075 - learning rate: 0.0001 - val loss: 0.568172416266273 at epoch: 18: 100%|█| 71 Loss: 0.5466463566657722 - learning rate: 0.0001 - val loss: 0.5592151249156279 at epoch: 19: 100%|█| Loss: 0.5339792922721894 - learning rate: 0.0001 - val loss: 0.5592151249156279 at epoch: 20: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000020.pth Loss: 0.533931184305473 - learning rate: 0.0001 - val loss: 0.6127981701317955 at epoch: 20: 100%|█| Loss: 0.5249322705189055 - learning rate: 0.0001 - val loss: 0.5131637243663564 at epoch: 21: 100%|█| Loss: 0.5161000969436544 - learning rate: 0.0001 - val loss: 0.5203708129770616 at epoch: 22: 100%|█| Loss: 0.5090960334866949 - learning rate: 0.0001 - val loss: 0.48682357107891755 at epoch: 23: 100%|█ Loss: 0.5065931637499954 - learning rate: 0.0001 - val loss: 0.5037144615369684 at epoch: 24: 100%|█| Loss: 0.502112655041937 - learning rate: 0.0001 - val loss: 0.5037144615369684 at epoch: 25: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000025.pth Loss: 0.5021178523283789 - learning rate: 0.0001 - val loss: 0.485143698313657 at epoch: 25: 100%|█| Loss: 0.49395890506906653 - learning rate: 0.0001 - val loss: 0.5013865723329431 at epoch: 26: 100%|█ Loss: 0.4848401721052545 - learning rate: 0.0001 - val loss: 0.4662446151761448 at epoch: 27: 100%|█| Loss: 0.47855537859632047 - learning rate: 0.0001 - val loss: 0.46448684965862946 at epoch: 28: 100%| Loss: 0.47601954828080106 - learning rate: 0.0001 - val loss: 0.4636546881759868 at epoch: 29: 100%|█ Loss: 0.46970804585091896 - learning rate: 0.0001 - val loss: 0.4636546881759868 at epoch: 30: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000030.pth Loss: 0.4699182764755632 - learning rate: 0.0001 - val loss: 0.4619622212999007 at epoch: 30: 100%|█| Loss: 0.4648659408840674 - learning rate: 0.0001 - val loss: 0.4609289923134972 at epoch: 31: 100%|█| Loss: 0.45938822793328776 - learning rate: 0.0001 - val loss: 0.4740794248440686 at epoch: 32: 100%|█ Loss: 0.4599477379807535 - learning rate: 0.0001 - val loss: 0.4809425420620862 at epoch: 33: 100%|█| Loss: 0.45654312175188105 - learning rate: 0.0001 - val loss: 0.45326191537520466 at epoch: 34: 100%| Loss: 0.45342455105075624 - learning rate: 0.0001 - val loss: 0.45326191537520466 at epoch: 35: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000035.pth Loss: 0.45325286767472783 - learning rate: 0.0001 - val loss: 0.47276994235375347 at epoch: 35: 100%| Loss: 0.450026126627809 - learning rate: 0.0001 - val loss: 0.4686397366663989 at epoch: 36: 100%|█| Loss: 0.4464021891230819 - learning rate: 0.0001 - val loss: 0.45992140033665824 at epoch: 37: 100%|█ Loss: 0.44005519808419413 - learning rate: 0.0001 - val loss: 0.440639721996644 at epoch: 38: 100%|█| Loss: 0.4380961273338339 - learning rate: 0.0001 - val loss: 0.4316103195442873 at epoch: 39: 100%|█| Loss: 0.43915227126309325 - learning rate: 0.0001 - val loss: 0.4316103195442873 at epoch: 40: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000040.pth Loss: 0.43948496672398873 - learning rate: 0.0001 - val loss: 0.4332115492400001 at epoch: 40: 100%|█ Loss: 0.434869290852979 - learning rate: 0.0001 - val loss: 0.4537683076718274 at epoch: 41: 100%|█| Loss: 0.4278905222106678 - learning rate: 0.0001 - val loss: 0.41909335816607757 at epoch: 42: 100%|█ Loss: 0.43464335021946054 - learning rate: 0.0001 - val loss: 0.4331094114219441 at epoch: 43: 100%|█ Loss: 0.4268188434165891 - learning rate: 0.0001 - val loss: 0.4238515408600078 at epoch: 44: 100%|█| Loss: 0.4210699434969678 - learning rate: 0.0001 - val loss: 0.4238515408600078 at epoch: 45: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000045.pth Loss: 0.4211959919088223 - learning rate: 0.0001 - val loss: 0.42071713594829335 at epoch: 45: 100%|█ Loss: 0.4237418758736162 - learning rate: 0.0001 - val loss: 0.4431636684081134 at epoch: 46: 100%|█| Loss: 0.41622830285188045 - learning rate: 0.0001 - val loss: 0.4005160454441519 at epoch: 47: 100%|█ Loss: 0.4174851571083734 - learning rate: 0.0001 - val loss: 0.4179435389883378 at epoch: 48: 100%|█| Loss: 0.4172716616669792 - learning rate: 0.0001 - val loss: 0.4095070677645066 at epoch: 49: 100%|█| Loss: 0.41583427957482844 - learning rate: 0.0001 - val loss: 0.4095070677645066 at epoch: 50: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000050.pth Loss: 0.4156778309302017 - learning rate: 0.0001 - val loss: 0.41162291519782124 at epoch: 50: 100%|█ Loss: 0.41320431664731544 - learning rate: 0.0001 - val loss: 0.42958581798216877 at epoch: 51: 100%| Loss: 0.41230651324762957 - learning rate: 0.0001 - val loss: 0.38863546532743115 at epoch: 52: 100%| Loss: 0.4107196634866536 - learning rate: 0.0001 - val loss: 0.42489026925143075 at epoch: 53: 100%|█ Loss: 0.40585786544794483 - learning rate: 0.0001 - val loss: 0.3980054452138789 at epoch: 54: 100%|█ Loss: 0.4039612961464754 - learning rate: 0.0001 - val loss: 0.3980054452138789 at epoch: 55: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000055.pth Loss: 0.40400318470958885 - learning rate: 0.0001 - val loss: 0.3978138250463149 at epoch: 55: 100%|█ Loss: 0.4049618319819994 - learning rate: 0.0001 - val loss: 0.40917643378762636 at epoch: 56: 100%|█ Loss: 0.4007464385639507 - learning rate: 0.0001 - val loss: 0.38810308189953074 at epoch: 57: 100%|█ Loss: 0.3985109680866098 - learning rate: 0.0001 - val loss: 0.3736323661663953 at epoch: 58: 100%|█| Loss: 0.40083664098710364 - learning rate: 0.0001 - val loss: 0.40081219638095184 at epoch: 59: 100%| Loss: 0.3977955708790092 - learning rate: 0.0001 - val loss: 0.40081219638095184 at epoch: 60: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000060.pth Loss: 0.39780675425522827 - learning rate: 0.0001 - val loss: 0.4037880213821636 at epoch: 60: 100%|█ Loss: 0.3947395866469193 - learning rate: 0.0001 - val loss: 0.39472170612391305 at epoch: 61: 100%|█ Loss: 0.3960546641253361 - learning rate: 0.0001 - val loss: 0.3994171093491947 at epoch: 62: 100%|█| Loss: 0.3994179342248616 - learning rate: 0.0001 - val loss: 0.3968759091461406 at epoch: 63: 100%|█| Loss: 0.39311319232153263 - learning rate: 0.0001 - val loss: 0.4006752617218915 at epoch: 64: 100%|█ Loss: 0.387932076342612 - learning rate: 0.0001 - val loss: 0.4006752617218915 at epoch: 65: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000065.pth Loss: 0.38790877867942075 - learning rate: 0.0001 - val loss: 0.4133554434075075 at epoch: 65: 100%|█ Loss: 0.3932054582125971 - learning rate: 0.0001 - val loss: 0.3904358411536497 at epoch: 66: 100%|█| Loss: 0.38771996388375507 - learning rate: 0.0001 - val loss: 0.36515841413946715 at epoch: 67: 100%| Loss: 0.3914431752104473 - learning rate: 0.0001 - val loss: 0.4122841708800372 at epoch: 68: 100%|█| Loss: 0.3854125591834054 - learning rate: 0.0001 - val loss: 0.371323906323489 at epoch: 69: 100%|█| Loss: 0.3845594668937795 - learning rate: 0.0001 - val loss: 0.371323906323489 at epoch: 70: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000070.pth Loss: 0.38447392668517893 - learning rate: 0.0001 - val loss: 0.4026658114264993 at epoch: 70: 100%|█ Loss: 0.3848543026623533 - learning rate: 0.0001 - val loss: 0.3795255545307608 at epoch: 71: 100%|█| Loss: 0.3831122824753345 - learning rate: 0.0001 - val loss: 0.3858378108809976 at epoch: 72: 100%|█| Loss: 0.3818983071103089 - learning rate: 0.0001 - val loss: 0.39288316579426036 at epoch: 73: 100%|█ Loss: 0.3801079845079318 - learning rate: 0.0001 - val loss: 0.38747620933196125 at epoch: 74: 100%|█ Loss: 0.38191624851533157 - learning rate: 0.0001 - val loss: 0.38747620933196125 at epoch: 75: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000075.pth Loss: 0.3819815127420625 - learning rate: 0.0001 - val loss: 0.36888961055699515 at epoch: 75: 100%|█ Loss: 0.3792302764567704 - learning rate: 0.0001 - val loss: 0.363033264875412 at epoch: 76: 100%|█| Loss: 0.3787335364805272 - learning rate: 0.0001 - val loss: 0.38272398534943075 at epoch: 77: 100%|█ Loss: 0.3747828765311168 - learning rate: 0.0001 - val loss: 0.37849925721392913 at epoch: 78: 100%|█ Loss: 0.3727061307496772 - learning rate: 0.0001 - val loss: 0.3726185192080105 at epoch: 79: 100%|█| Loss: 0.3749956050315383 - learning rate: 0.0001 - val loss: 0.3726185192080105 at epoch: 80: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000080.pth Loss: 0.37504371609980425 - learning rate: 0.0001 - val loss: 0.37050309952567606 at epoch: 80: 100%| Loss: 0.3747331331298275 - learning rate: 0.0001 - val loss: 0.36530383895425234 at epoch: 81: 100%|█ Loss: 0.37707984173148246 - learning rate: 0.0001 - val loss: 0.35689589381217957 at epoch: 82: 100%| Loss: 0.3732697500677122 - learning rate: 0.0001 - val loss: 0.3590216654188493 at epoch: 83: 100%|█| Loss: 0.3694498129552711 - learning rate: 0.0001 - val loss: 0.36992430862258463 at epoch: 84: 100%|█ Loss: 0.37053039309948516 - learning rate: 0.0001 - val loss: 0.36992430862258463 at epoch: 85: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000085.pth Loss: 0.3704947035557057 - learning rate: 0.0001 - val loss: 0.38330086013850045 at epoch: 85: 100%|█ Loss: 0.36746956360223737 - learning rate: 0.0001 - val loss: 0.3864254951477051 at epoch: 86: 100%|█ Loss: 0.3703802171029473 - learning rate: 0.0001 - val loss: 0.3766837190179264 at epoch: 87: 100%|█| Loss: 0.36902527093388543 - learning rate: 0.0001 - val loss: 0.37262428332777586 at epoch: 88: 100%| Loss: 0.3665164634273641 - learning rate: 0.0001 - val loss: 0.3632035798886243 at epoch: 89: 100%|█| Loss: 0.3732048325948209 - learning rate: 0.0001 - val loss: 0.3632035798886243 at epoch: 90: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000090.pth Loss: 0.373401274283727 - learning rate: 0.0001 - val loss: 0.37565673624767976 at epoch: 90: 100%|█| Loss: 0.36413453232543735 - learning rate: 0.0001 - val loss: 0.36815528133336234 at epoch: 91: 100%| Loss: 0.36397053827135467 - learning rate: 0.0001 - val loss: 0.3481050168766695 at epoch: 92: 100%|█ Loss: 0.3620503853845463 - learning rate: 0.0001 - val loss: 0.3631353991873124 at epoch: 93: 100%|█| Loss: 0.3647868237776404 - learning rate: 0.0001 - val loss: 0.3595329803578994 at epoch: 94: 100%|█| Loss: 0.35972402996107855 - learning rate: 0.0001 - val loss: 0.3595329803578994 at epoch: 95: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000095.pth Loss: 0.3598175739019984 - learning rate: 0.0001 - val loss: 0.368411181604161 at epoch: 95: 100%|█| Loss: 0.3618277128853392 - learning rate: 0.0001 - val loss: 0.37604328113443714 at epoch: 96: 100%|█ Loss: 0.3625654221162969 - learning rate: 0.0001 - val loss: 0.3826728410580579 at epoch: 97: 100%|█| Loss: 0.359980691094445 - learning rate: 0.0001 - val loss: 0.35401028745314655 at epoch: 98: 100%|█| Loss: 0.3589852239879438 - learning rate: 0.0001 - val loss: 0.3632833396687227 at epoch: 99: 100%|█| Loss: 0.3591972673155742 - learning rate: 0.0001 - val loss: 0.3632833396687227 at epoch: 100: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000100.pth Loss: 0.3589290365098743 - learning rate: 0.0001 - val loss: 0.35630277149817524 at epoch: 100: 100%| Loss: 0.35614064462597234 - learning rate: 0.0001 - val loss: 0.3536679411635679 at epoch: 101: 100%| Loss: 0.35828276323557234 - learning rate: 0.0001 - val loss: 0.3508167740176706 at epoch: 102: 100%| Loss: 0.3540584946485078 - learning rate: 0.0001 - val loss: 0.35323924702756543 at epoch: 103: 100%| Loss: 0.35430986643086915 - learning rate: 0.0001 - val loss: 0.34429293870925903 at epoch: 104: 100% Loss: 0.3577814444108715 - learning rate: 0.0001 - val loss: 0.34429293870925903 at epoch: 105: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000105.pth Loss: 0.3578635099417994 - learning rate: 0.0001 - val loss: 0.3769519854994381 at epoch: 105: 100%|█ Loss: 0.3533500170898903 - learning rate: 0.0001 - val loss: 0.36215731326271505 at epoch: 106: 100%| Loss: 0.356513929583205 - learning rate: 0.0001 - val loss: 0.36290551809703603 at epoch: 107: 100%|█ Loss: 0.3555090062247493 - learning rate: 0.0001 - val loss: 0.36206039611031027 at epoch: 108: 100%| Loss: 0.35563033210266914 - learning rate: 0.0001 - val loss: 0.34595397114753723 at epoch: 109: 100% Loss: 0.35197701686004684 - learning rate: 0.0001 - val loss: 0.34595397114753723 at epoch: 110: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000110.pth Loss: 0.35189396344551316 - learning rate: 0.0001 - val loss: 0.3333049837280722 at epoch: 110: 100%| Loss: 0.35038518548178044 - learning rate: 0.0001 - val loss: 0.36820768959382 at epoch: 111: 100%|█| Loss: 0.35086454985198284 - learning rate: 0.0001 - val loss: 0.3493247926235199 at epoch: 112: 100%| Loss: 0.3508417098633581 - learning rate: 0.0001 - val loss: 0.3344658227527843 at epoch: 113: 100%|█ Loss: 0.3499371398193567 - learning rate: 0.0001 - val loss: 0.3559708349844989 at epoch: 114: 100%|█ Loss: 0.3488510699953114 - learning rate: 0.0001 - val loss: 0.3559708349844989 at epoch: 115: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000115.pth Loss: 0.34897961025873153 - learning rate: 0.0001 - val loss: 0.34428879969260273 at epoch: 115: 100% Loss: 0.34789785368671977 - learning rate: 0.0001 - val loss: 0.3504102580687579 at epoch: 116: 100%| Loss: 0.34687935713694185 - learning rate: 0.0001 - val loss: 0.34937093450742607 at epoch: 117: 100% Loss: 0.34958091423933807 - learning rate: 0.0001 - val loss: 0.35065576697097106 at epoch: 118: 100% Loss: 0.34731799226924465 - learning rate: 0.0001 - val loss: 0.3686371077509487 at epoch: 119: 100%| Loss: 0.35628663659012516 - learning rate: 0.0001 - val loss: 0.3686371077509487 at epoch: 120: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000120.pth Loss: 0.3565249024699423 - learning rate: 0.0001 - val loss: 0.3400667485068826 at epoch: 120: 100%|█ Loss: 0.3487435434860498 - learning rate: 0.0001 - val loss: 0.3453256154761595 at epoch: 121: 100%|█ Loss: 0.3515215716642981 - learning rate: 0.0001 - val loss: 0.32869693636894226 at epoch: 122: 100%| Loss: 0.35017006837008224 - learning rate: 0.0001 - val loss: 0.34082086997873645 at epoch: 123: 100% Loss: 0.34229803519684565 - learning rate: 0.0001 - val loss: 0.3332931294160731 at epoch: 124: 100%| Loss: 0.34396075596772757 - learning rate: 0.0001 - val loss: 0.3332931294160731 at epoch: 125: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000125.pth Loss: 0.3442547787058802 - learning rate: 0.0001 - val loss: 0.349171065232333 at epoch: 125: 100%|█| Loss: 0.34577042236404604 - learning rate: 0.0001 - val loss: 0.3645528414670159 at epoch: 126: 100%| Loss: 0.34132711608040783 - learning rate: 0.0001 - val loss: 0.3360985079232384 at epoch: 127: 100%| Loss: 0.3437456882648388 - learning rate: 0.0001 - val loss: 0.3382779079325059 at epoch: 128: 100%|█ Loss: 0.3492086415427167 - learning rate: 0.0001 - val loss: 0.34212653952486377 at epoch: 129: 100%| Loss: 0.3454139971450054 - learning rate: 0.0001 - val loss: 0.34212653952486377 at epoch: 130: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000130.pth Loss: 0.3456618153649893 - learning rate: 0.0001 - val loss: 0.3411855013931499 at epoch: 130: 100%|█ Loss: 0.34195975246456045 - learning rate: 0.0001 - val loss: 0.3484768604531008 at epoch: 131: 100%| Loss: 0.3413154762368156 - learning rate: 0.0001 - val loss: 0.3232783622601453 at epoch: 132: 100%|█ Loss: 0.34463959639291886 - learning rate: 0.0001 - val loss: 0.3466600670534022 at epoch: 133: 100%| Loss: 0.3417269390081594 - learning rate: 0.0001 - val loss: 0.3469894247896531 at epoch: 134: 100%|█ Loss: 0.34211664921578083 - learning rate: 0.0001 - val loss: 0.3469894247896531 at epoch: 135: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000135.pth Loss: 0.34235621625076945 - learning rate: 0.0001 - val loss: 0.32455766726942625 at epoch: 135: 100% Loss: 0.3417684130755096 - learning rate: 0.0001 - val loss: 0.34222139155163483 at epoch: 136: 100%| Loss: 0.34200943588711724 - learning rate: 0.0001 - val loss: 0.340353382860913 at epoch: 137: 100%|█ Loss: 0.3427103012176737 - learning rate: 0.0001 - val loss: 0.31848034700926614 at epoch: 138: 100%| Loss: 0.3389667235782456 - learning rate: 0.0001 - val loss: 0.33617665311869455 at epoch: 139: 100%| Loss: 0.3409421077861799 - learning rate: 0.0001 - val loss: 0.33617665311869455 at epoch: 140: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000140.pth Loss: 0.341106344279385 - learning rate: 0.0001 - val loss: 0.3356095131705789 at epoch: 140: 100%|█| Loss: 0.33813529564496364 - learning rate: 0.0001 - val loss: 0.3460811920025769 at epoch: 141: 100%| Loss: 0.33777565426989414 - learning rate: 0.0001 - val loss: 0.3389185053460738 at epoch: 142: 100%| Loss: 0.33634137785168705 - learning rate: 0.0001 - val loss: 0.33260248426128836 at epoch: 143: 100% Loss: 0.33574531278267733 - learning rate: 0.0001 - val loss: 0.3235555492779788 at epoch: 144: 100%| Loss: 0.33875305039090153 - learning rate: 0.0001 - val loss: 0.3235555492779788 at epoch: 145: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000145.pth Loss: 0.33885545482030305 - learning rate: 0.0001 - val loss: 0.35587654569569754 at epoch: 145: 100% Loss: 0.33802067140322517 - learning rate: 0.0001 - val loss: 0.33407504768932567 at epoch: 146: 100% Loss: 0.3392781635624262 - learning rate: 0.0001 - val loss: 0.3666040161076714 at epoch: 147: 100%|█ Loss: 0.3371390947653825 - learning rate: 0.0001 - val loss: 0.3289692428182153 at epoch: 148: 100%|█ Loss: 0.33789917689487026 - learning rate: 0.0001 - val loss: 0.3505098136032329 at epoch: 149: 100%| Loss: 0.3346090515940216 - learning rate: 0.0001 - val loss: 0.3505098136032329 at epoch: 150: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000150.pth Loss: 0.33460028960033106 - learning rate: 0.0001 - val loss: 0.33348899378496055 at epoch: 150: 100% Loss: 0.3352192599570568 - learning rate: 0.0001 - val loss: 0.34297194726326885 at epoch: 151: 100%| Loss: 0.33863768197668814 - learning rate: 0.0001 - val loss: 0.33424725953270407 at epoch: 152: 100% Loss: 0.33613811283680184 - learning rate: 0.0001 - val loss: 0.3319559430374819 at epoch: 153: 100%| Loss: 0.3353894404453879 - learning rate: 0.0001 - val loss: 0.33400768216918497 at epoch: 154: 100%| Loss: 0.33209147945296164 - learning rate: 0.0001 - val loss: 0.33400768216918497 at epoch: 155: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000155.pth Loss: 0.3320029157225557 - learning rate: 0.0001 - val loss: 0.34071629012332244 at epoch: 155: 100%| Loss: 0.33204671484905307 - learning rate: 0.0001 - val loss: 0.31705038161838756 at epoch: 156: 100% Loss: 0.33419463572558666 - learning rate: 0.0001 - val loss: 0.3177177625543931 at epoch: 157: 100%| Loss: 0.3301593000419968 - learning rate: 0.0001 - val loss: 0.3259129383984734 at epoch: 158: 100%|█ Loss: 0.3312155745476692 - learning rate: 0.0001 - val loss: 0.34573231549823985 at epoch: 159: 100%| Loss: 0.33380665605128146 - learning rate: 0.0001 - val loss: 0.34573231549823985 at epoch: 160: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000160.pth Loss: 0.33404608580025336 - learning rate: 0.0001 - val loss: 0.3204050204333137 at epoch: 160: 100%| Loss: 0.33061359949524266 - learning rate: 0.0001 - val loss: 0.3419087257455377 at epoch: 161: 100%| Loss: 0.3285083958003644 - learning rate: 0.0001 - val loss: 0.32915519440875335 at epoch: 162: 100%| Loss: 0.327556164568937 - learning rate: 0.0001 - val loss: 0.3519600980422076 at epoch: 163: 100%|█| Loss: 0.3347715123802049 - learning rate: 0.0001 - val loss: 0.3165829830309924 at epoch: 164: 100%|█ Loss: 0.33297615071664977 - learning rate: 0.0001 - val loss: 0.3165829830309924 at epoch: 165: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000165.pth Loss: 0.3331423436955262 - learning rate: 0.0001 - val loss: 0.3135868030435899 at epoch: 165: 100%|█ Loss: 0.33309593017380607 - learning rate: 0.0001 - val loss: 0.32883938270456653 at epoch: 166: 100% Loss: 0.33548486739854266 - learning rate: 0.0001 - val loss: 0.33917106074445386 at epoch: 167: 100% Loss: 0.3376981250453361 - learning rate: 0.0001 - val loss: 0.33148147779352527 at epoch: 168: 100%| Loss: 0.3325687631071029 - learning rate: 0.0001 - val loss: 0.33081743296454935 at epoch: 169: 100%| Loss: 0.3324487292358662 - learning rate: 0.0001 - val loss: 0.33081743296454935 at epoch: 170: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000170.pth Loss: 0.33248443889069257 - learning rate: 0.0001 - val loss: 0.3290778459871517 at epoch: 170: 100%| Loss: 0.32513701452287835 - learning rate: 0.0001 - val loss: 0.348345291965148 at epoch: 171: 100%|█ Loss: 0.32839437105499386 - learning rate: 0.0001 - val loss: 0.3147103365729837 at epoch: 172: 100%| Loss: 0.3292263769122513 - learning rate: 0.0001 - val loss: 0.3235371025169597 at epoch: 173: 100%|█ Loss: 0.33282532101312606 - learning rate: 0.0001 - val loss: 0.3237471466555315 at epoch: 174: 100%| Loss: 0.3292505520099368 - learning rate: 0.0001 - val loss: 0.3237471466555315 at epoch: 175: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000175.pth Loss: 0.32915210520206967 - learning rate: 0.0001 - val loss: 0.32510341616237864 at epoch: 175: 100% Loss: 0.32848219482054103 - learning rate: 0.0001 - val loss: 0.31360289016190696 at epoch: 176: 100% Loss: 0.32775541570562367 - learning rate: 0.0001 - val loss: 0.32836197754916024 at epoch: 177: 100% Loss: 0.32911148453274197 - learning rate: 0.0001 - val loss: 0.33613835538134856 at epoch: 178: 100% Loss: 0.3283213041359078 - learning rate: 0.0001 - val loss: 0.3272248716915355 at epoch: 179: 100%|█ Loss: 0.32704727372477177 - learning rate: 0.0001 - val loss: 0.3272248716915355 at epoch: 180: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000180.pth Loss: 0.3272566455760594 - learning rate: 0.0001 - val loss: 0.33723972124211926 at epoch: 180: 100%| Loss: 0.3257023029372283 - learning rate: 0.0001 - val loss: 0.32358309889540954 at epoch: 181: 100%| Loss: 0.3287189950961282 - learning rate: 0.0001 - val loss: 0.31701383871190686 at epoch: 182: 100%| Loss: 0.33806383744858964 - learning rate: 0.0001 - val loss: 0.33990464315694924 at epoch: 183: 100% Loss: 0.3448227711718119 - learning rate: 0.0001 - val loss: 0.323043777662165 at epoch: 184: 100%|█| Loss: 0.32528840119862024 - learning rate: 0.0001 - val loss: 0.323043777662165 at epoch: 185: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000185.pth Loss: 0.32527250814221725 - learning rate: 0.0001 - val loss: 0.31601231238421273 at epoch: 185: 100% Loss: 0.32251234111509897 - learning rate: 0.0001 - val loss: 0.30739615682293386 at epoch: 186: 100% Loss: 0.3262497771692409 - learning rate: 0.0001 - val loss: 0.3419104832060197 at epoch: 187: 100%|█ Loss: 0.3231987905385936 - learning rate: 0.0001 - val loss: 0.3456746129428639 at epoch: 188: 100%|█ Loss: 0.32693653987042576 - learning rate: 0.0001 - val loss: 0.32475994790301604 at epoch: 189: 100% Loss: 0.32213649243199627 - learning rate: 0.0001 - val loss: 0.32475994790301604 at epoch: 190: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000190.pth Loss: 0.32206250203453846 - learning rate: 0.0001 - val loss: 0.32578406614415784 at epoch: 190: 100% Loss: 0.33468802707630885 - learning rate: 0.0001 - val loss: 0.32397622045348673 at epoch: 191: 100% Loss: 0.32182675868148086 - learning rate: 0.0001 - val loss: 0.34383298193707185 at epoch: 192: 100% Loss: 0.32288886045561677 - learning rate: 0.0001 - val loss: 0.31198186909451203 at epoch: 193: 100% Loss: 0.3184754801927085 - learning rate: 0.0001 - val loss: 0.32010549569831176 at epoch: 194: 100%| Loss: 0.3190527330646968 - learning rate: 0.0001 - val loss: 0.32010549569831176 at epoch: 195: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000195.pth Loss: 0.3188870509714927 - learning rate: 0.0001 - val loss: 0.31590445953256946 at epoch: 195: 100%| Loss: 0.31914277071317704 - learning rate: 0.0001 - val loss: 0.32708724106059356 at epoch: 196: 100% Loss: 0.31704306400886306 - learning rate: 0.0001 - val loss: 0.31749794062446146 at epoch: 197: 100% Loss: 0.31728993576523984 - learning rate: 0.0001 - val loss: 0.3153178534087013 at epoch: 198: 100%| Loss: 0.32232077654601804 - learning rate: 0.0001 - val loss: 0.3280895422486698 at epoch: 199: 100%| Loss: 0.3228824969027295 - learning rate: 0.0001 - val loss: 0.3280895422486698 at epoch: 200: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000200.pth Loss: 0.32287279868724456 - learning rate: 0.0001 - val loss: 0.4071896163856282 at epoch: 200: 100%| Loss: 0.3160588775386039 - learning rate: 0.0001 - val loss: 0.30915284682722655 at epoch: 201: 100%| Loss: 0.3210019682491175 - learning rate: 0.0001 - val loss: 0.31598322356448455 at epoch: 202: 100%| Loss: 0.32000632737886625 - learning rate: 0.0001 - val loss: 0.31841721604852113 at epoch: 203: 100% Loss: 0.32569978021428175 - learning rate: 0.0001 - val loss: 0.30576730212744546 at epoch: 204: 100% Loss: 0.3216656012741547 - learning rate: 0.0001 - val loss: 0.30576730212744546 at epoch: 205: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000205.pth Loss: 0.32162326989978596 - learning rate: 0.0001 - val loss: 0.4041090800481684 at epoch: 205: 100%| Loss: 0.3229070930135134 - learning rate: 0.0001 - val loss: 0.31858688417602987 at epoch: 206: 100%| Loss: 0.3147415189147827 - learning rate: 0.0001 - val loss: 0.3235223661450779 at epoch: 207: 100%|█ Loss: 0.32002344245358655 - learning rate: 0.0001 - val loss: 0.3174469821593341 at epoch: 208: 100%| Loss: 0.3156113898488102 - learning rate: 0.0001 - val loss: 0.31989597047076507 at epoch: 209: 100%| Loss: 0.32095162807729655 - learning rate: 0.0001 - val loss: 0.31989597047076507 at epoch: 210: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000210.pth Loss: 0.32093586137770275 - learning rate: 0.0001 - val loss: 0.33956563122132244 at epoch: 210: 100% Loss: 0.3160227767219104 - learning rate: 0.0001 - val loss: 0.32169514719177694 at epoch: 211: 100%| Loss: 0.3175911024067359 - learning rate: 0.0001 - val loss: 0.3318972885608673 at epoch: 212: 100%|█ Loss: 0.31850152476229593 - learning rate: 0.0001 - val loss: 0.3349080576616175 at epoch: 213: 100%| Loss: 0.3189213678677212 - learning rate: 0.0001 - val loss: 0.3824808729045531 at epoch: 214: 100%|█ Loss: 0.32222099435729024 - learning rate: 0.0001 - val loss: 0.3824808729045531 at epoch: 215: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000215.pth Loss: 0.3225129536837547 - learning rate: 0.0001 - val loss: 0.32496700041434345 at epoch: 215: 100%| Loss: 0.3163468918166899 - learning rate: 0.0001 - val loss: 0.3155099679442013 at epoch: 216: 100%|█ Loss: 0.3181705278041805 - learning rate: 0.0001 - val loss: 0.2923184168689391 at epoch: 217: 100%|█ Loss: 0.31520228794512223 - learning rate: 0.0001 - val loss: 0.32073016902979684 at epoch: 218: 100% Loss: 0.31328052384168364 - learning rate: 0.0001 - val loss: 0.3084256368524888 at epoch: 219: 100%| Loss: 0.314738614016405 - learning rate: 0.0001 - val loss: 0.3084256368524888 at epoch: 220: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000220.pth Loss: 0.31483178794633393 - learning rate: 0.0001 - val loss: 0.4059099555015564 at epoch: 220: 100%| Loss: 0.3251883854551794 - learning rate: 0.0001 - val loss: 0.31899913181276884 at epoch: 221: 100%| Loss: 0.317881585746962 - learning rate: 0.0001 - val loss: 0.31094854719498577 at epoch: 222: 100%|█ Loss: 0.3171200509932583 - learning rate: 0.0001 - val loss: 0.3139798185404609 at epoch: 223: 100%|█ Loss: 0.3164741094100758 - learning rate: 0.0001 - val loss: 0.3251757253618801 at epoch: 224: 100%|█ Loss: 0.31757013108477244 - learning rate: 0.0001 - val loss: 0.3251757253618801 at epoch: 225: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000225.pth Loss: 0.31753265616617776 - learning rate: 0.0001 - val loss: 0.3073840264011832 at epoch: 225: 100%| Loss: 0.3136430283803155 - learning rate: 0.0001 - val loss: 0.32858657661606283 at epoch: 226: 100%| Loss: 0.31154451102558706 - learning rate: 0.0001 - val loss: 0.3088152750450022 at epoch: 227: 100%| Loss: 0.3164677088521681 - learning rate: 0.0001 - val loss: 0.331315336858525 at epoch: 228: 100%|█| Loss: 0.31428529164199215 - learning rate: 0.0001 - val loss: 0.3225905772517709 at epoch: 229: 100%| Loss: 0.3151915768088575 - learning rate: 0.0001 - val loss: 0.3225905772517709 at epoch: 230: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000230.pth Loss: 0.31526857360968863 - learning rate: 0.0001 - val loss: 0.3183743234942941 at epoch: 230: 100%| Loss: 0.3171861544771008 - learning rate: 0.0001 - val loss: 0.3250532816438114 at epoch: 231: 100%|█ Loss: 0.3126005499806863 - learning rate: 0.0001 - val loss: 0.31594983093878803 at epoch: 232: 100%| Loss: 0.31241105771214395 - learning rate: 0.0001 - val loss: 0.3324458371190464 at epoch: 233: 100%| Loss: 0.30969504739782633 - learning rate: 0.0001 - val loss: 0.3988265027018154 at epoch: 234: 100%| Loss: 0.3163540782667072 - learning rate: 0.0001 - val loss: 0.3988265027018154 at epoch: 235: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000235.pth Loss: 0.31643667399384484 - learning rate: 0.0001 - val loss: 0.3087838818045223 at epoch: 235: 100%| Loss: 0.31542253953559296 - learning rate: 0.0001 - val loss: 0.3142048400991103 at epoch: 236: 100%| Loss: 0.3157057684418879 - learning rate: 0.0001 - val loss: 0.32353735145400553 at epoch: 237: 100%| Loss: 0.3120209511520141 - learning rate: 0.0001 - val loss: 0.31309520847657146 at epoch: 238: 100%| Loss: 0.30913533819021044 - learning rate: 0.0001 - val loss: 0.3053393074694802 at epoch: 239: 100%| Loss: 0.3087995447135171 - learning rate: 0.0001 - val loss: 0.3053393074694802 at epoch: 240: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000240.pth Loss: 0.3087463332560937 - learning rate: 0.0001 - val loss: 0.3060222487239277 at epoch: 240: 100%|█ Loss: 0.31536922673467477 - learning rate: 0.0001 - val loss: 0.30904537176384644 at epoch: 241: 100% Loss: 0.31636830333053984 - learning rate: 0.0001 - val loss: 0.3328818792805952 at epoch: 242: 100%| Loss: 0.31180857620728064 - learning rate: 0.0001 - val loss: 0.32575685487074013 at epoch: 243: 100% Loss: 0.30974811510893424 - learning rate: 0.0001 - val loss: 0.309741834507269 at epoch: 244: 100%|█ Loss: 0.3107690839877342 - learning rate: 0.0001 - val loss: 0.309741834507269 at epoch: 245: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000245.pth Loss: 0.31067151745659205 - learning rate: 0.0001 - val loss: 0.3386328009998097 at epoch: 245: 100%| Loss: 0.3123398712281569 - learning rate: 0.0001 - val loss: 0.29813522801679726 at epoch: 246: 100%| Loss: 0.3130464750852545 - learning rate: 0.0001 - val loss: 0.31816019906717186 at epoch: 247: 100%| Loss: 0.3116480052221769 - learning rate: 0.0001 - val loss: 0.33110381224576163 at epoch: 248: 100%| Loss: 0.30647446732640765 - learning rate: 0.0001 - val loss: 0.4002171351629145 at epoch: 249: 100%| Loss: 0.30593346250789794 - learning rate: 0.0001 - val loss: 0.4002171351629145 at epoch: 250: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000250.pth Loss: 0.30605353545943065 - learning rate: 0.0001 - val loss: 0.3236594989019282 at epoch: 250: 100%| Loss: 0.31523266081364415 - learning rate: 0.0001 - val loss: 0.31214223188512463 at epoch: 251: 100% Loss: 0.3121627108239396 - learning rate: 0.0001 - val loss: 0.32304346386124105 at epoch: 252: 100%| Loss: 0.31201716769107385 - learning rate: 0.0001 - val loss: 0.40594975737964406 at epoch: 253: 100% Loss: 0.3132015399354272 - learning rate: 0.0001 - val loss: 0.29918953425744 at epoch: 254: 100%|█| Loss: 0.30866048826352177 - learning rate: 0.0001 - val loss: 0.29918953425744 at epoch: 255: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000255.pth Loss: 0.3088879721600308 - learning rate: 0.0001 - val loss: 0.3033159322598401 at epoch: 255: 100%|█ Loss: 0.30523368271491186 - learning rate: 0.0001 - val loss: 0.32445379215128284 at epoch: 256: 100% Loss: 0.3082592141304056 - learning rate: 0.0001 - val loss: 0.2987316995859146 at epoch: 257: 100%|█ Loss: 0.3077013487596392 - learning rate: 0.0001 - val loss: 0.32674457921701316 at epoch: 258: 100%| Loss: 0.31010557551098 - learning rate: 0.0001 - val loss: 0.3030623881255879 at epoch: 259: 100%|█| Loss: 0.3065339962239039 - learning rate: 0.0001 - val loss: 0.3030623881255879 at epoch: 260: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000260.pth Loss: 0.30648479753042696 - learning rate: 0.0001 - val loss: 0.2945685412953882 at epoch: 260: 100%| Loss: 0.3041540330465535 - learning rate: 0.0001 - val loss: 0.30476875252583446 at epoch: 261: 100%| Loss: 0.3068798369983915 - learning rate: 0.0001 - val loss: 0.31297055938664603 at epoch: 262: 100%| Loss: 0.30861558841029635 - learning rate: 0.0001 - val loss: 0.31001174186958985 at epoch: 263: 100% Loss: 0.31007959642752775 - learning rate: 0.0001 - val loss: 0.31483353060834546 at epoch: 264: 100% Loss: 0.307805921882391 - learning rate: 0.0001 - val loss: 0.31483353060834546 at epoch: 265: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000265.pth Loss: 0.3080047332218003 - learning rate: 0.0001 - val loss: 0.3080143954823999 at epoch: 265: 100%|█ Loss: 0.30392203399992057 - learning rate: 0.0001 - val loss: 0.29799437347580404 at epoch: 266: 100% Loss: 0.3053085238281819 - learning rate: 0.0001 - val loss: 0.3291468602769515 at epoch: 267: 100%|█ Loss: 0.30717476932240995 - learning rate: 5e-05 - val loss: 0.30195652737337 at epoch: 268: 100%|█| Loss: 0.3009409159214094 - learning rate: 5e-05 - val loss: 0.2819372073692434 at epoch: 269: 100%|█| Loss: 0.3001459057770628 - learning rate: 5e-05 - val loss: 0.2819372073692434 at epoch: 270: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000270.pth Loss: 0.3000148224074139 - learning rate: 5e-05 - val loss: 0.30267828615272746 at epoch: 270: 100%|█ Loss: 0.2992089938793222 - learning rate: 5e-05 - val loss: 0.30576135218143463 at epoch: 271: 100%|█ Loss: 0.297558757301155 - learning rate: 5e-05 - val loss: 0.31899135428316455 at epoch: 272: 100%|█| Loss: 0.297383855617861 - learning rate: 5e-05 - val loss: 0.30306536835782666 at epoch: 273: 100%|█| Loss: 0.2956093915503062 - learning rate: 5e-05 - val loss: 0.29790930537616506 at epoch: 274: 100%|█ Loss: 0.30059484900209493 - learning rate: 5e-05 - val loss: 0.29790930537616506 at epoch: 275: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000275.pth Loss: 0.30052305703964527 - learning rate: 5e-05 - val loss: 0.3061572313308716 at epoch: 275: 100%|█ Loss: 0.3003741754647578 - learning rate: 5e-05 - val loss: 0.29603279601125154 at epoch: 276: 100%|█ Loss: 0.299627613510048 - learning rate: 5e-05 - val loss: 0.28398248027352724 at epoch: 277: 100%|█| Loss: 0.3021080169301984 - learning rate: 5e-05 - val loss: 0.3111071569078109 at epoch: 278: 100%|█| Loss: 0.2939733659206906 - learning rate: 5e-05 - val loss: 0.2877907016698052 at epoch: 279: 100%|█| Loss: 0.298268929477677 - learning rate: 5e-05 - val loss: 0.2877907016698052 at epoch: 280: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000280.pth Loss: 0.2983553654188225 - learning rate: 5e-05 - val loss: 0.30654799675240235 at epoch: 280: 100%|█ Loss: 0.2983045185626468 - learning rate: 5e-05 - val loss: 0.3073072275694679 at epoch: 281: 100%|█| Loss: 0.2977428478052427 - learning rate: 5e-05 - val loss: 0.29910268678384666 at epoch: 282: 100%|█ Loss: 0.2954770494746031 - learning rate: 5e-05 - val loss: 0.3032520132906297 at epoch: 283: 100%|█| Loss: 0.2962064707919311 - learning rate: 5e-05 - val loss: 0.3157319461598116 at epoch: 284: 100%|█| Loss: 0.2965031956535811 - learning rate: 5e-05 - val loss: 0.3157319461598116 at epoch: 285: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000285.pth Loss: 0.29662289303133843 - learning rate: 5e-05 - val loss: 0.29873434760991263 at epoch: 285: 100%| Loss: 0.29265239134410104 - learning rate: 5e-05 - val loss: 0.32044096378719106 at epoch: 286: 100%| Loss: 0.29690624289266565 - learning rate: 5e-05 - val loss: 0.29064199416076436 at epoch: 287: 100%| Loss: 0.292572217319301 - learning rate: 5e-05 - val loss: 0.2932455557234147 at epoch: 288: 100%|█| Loss: 0.2910314104214539 - learning rate: 5e-05 - val loss: 0.3159342232872458 at epoch: 289: 100%|█| Loss: 0.2917812002104754 - learning rate: 5e-05 - val loss: 0.3159342232872458 at epoch: 290: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000290.pth Loss: 0.2919222831809205 - learning rate: 5e-05 - val loss: 0.3200300826745875 at epoch: 290: 100%|█| Loss: 0.2920384829554764 - learning rate: 5e-05 - val loss: 0.2998523282654145 at epoch: 291: 100%|█| Loss: 0.2966673892911009 - learning rate: 5e-05 - val loss: 0.2861257344484329 at epoch: 292: 100%|█| Loss: 0.29242345933884256 - learning rate: 5e-05 - val loss: 0.3010793976923999 at epoch: 293: 100%|█ Loss: 0.29116458153791175 - learning rate: 5e-05 - val loss: 0.29494749184917 at epoch: 294: 100%|█| Loss: 0.3043915748845932 - learning rate: 5e-05 - val loss: 0.29494749184917 at epoch: 295: 100%|▉| 7Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000295.pth Loss: 0.3045244020730714 - learning rate: 5e-05 - val loss: 0.29799241002868204 at epoch: 295: 100%|█ Loss: 0.29013394646481655 - learning rate: 5e-05 - val loss: 0.2721177050295998 at epoch: 296: 100%|█ Loss: 0.2957085180648367 - learning rate: 5e-05 - val loss: 0.2805605329134885 at epoch: 297: 100%|█| Loss: 0.29426202751602587 - learning rate: 5e-05 - val loss: 0.30168647012289834 at epoch: 298: 100%| Loss: 0.2920073376789253 - learning rate: 5e-05 - val loss: 0.2961472237811369 at epoch: 299: 100%|█| Loss: 0.2952373994087707 - learning rate: 5e-05 - val loss: 0.2961472237811369 at epoch: 300: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000300.pth Loss: 0.29522111856039596 - learning rate: 5e-05 - val loss: 0.28166491494459267 at epoch: 300: 100%| Loss: 0.29718377228145654 - learning rate: 5e-05 - val loss: 0.32492024057051716 at epoch: 301: 100%| Loss: 0.29203436547575967 - learning rate: 5e-05 - val loss: 0.29104062125963326 at epoch: 302: 100%| Loss: 0.29241493915829864 - learning rate: 5e-05 - val loss: 0.2915509176604888 at epoch: 303: 100%|█ Loss: 0.2910756430117156 - learning rate: 5e-05 - val loss: 0.3040669306236155 at epoch: 304: 100%|█| Loss: 0.2957818931665833 - learning rate: 5e-05 - val loss: 0.3040669306236155 at epoch: 305: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000305.pth Loss: 0.2959351596348455 - learning rate: 5e-05 - val loss: 0.3111670963904437 at epoch: 305: 100%|█| Loss: 0.29206380135843446 - learning rate: 5e-05 - val loss: 0.3021305343684028 at epoch: 306: 100%|█ Loss: 0.2906969288361455 - learning rate: 5e-05 - val loss: 0.30570460943614736 at epoch: 307: 100%|█ Loss: 0.2900724932447471 - learning rate: 5e-05 - val loss: 0.3324916222516228 at epoch: 308: 100%|█| Loss: 0.29348984774269654 - learning rate: 5e-05 - val loss: 0.3073321966563954 at epoch: 309: 100%|█ Loss: 0.2912182815622684 - learning rate: 5e-05 - val loss: 0.3073321966563954 at epoch: 310: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000310.pth Loss: 0.29140155960908326 - learning rate: 5e-05 - val loss: 0.3095227970796473 at epoch: 310: 100%|█ Loss: 0.28802871273900676 - learning rate: 5e-05 - val loss: 0.3084297224002726 at epoch: 311: 100%|█ Loss: 0.2929448010698688 - learning rate: 5e-05 - val loss: 0.3082868711036794 at epoch: 312: 100%|█

[Mayur28]

Hi @voducman ,

Thanks for getting back to me. Looking at the logs, I would presume that you are not training the model using the dataset that the authors trained their models (this is also the dataset that I am trying to train with, called LRS2). Could you please confirm whether you are using another dataset please. I thought that this is the case because the model converges much quicker than expected, and I see that, from your checkpoint name, you are training with 192x192 images whereas the original implementation uses 96x96. I am surprised that your model converges really quickly - when reading through the issues, the authors state that the loss stays around 0.68 - 0.69 for a very long time, and at around 150K -200K steps, the model begins to converge and apparently reaches full convergence at around 700K steps.

[voducman]

Hi @Mayur28 , I am using LRS2 to train for my model. I modified this original model to adapt with a new large size is 192x192. I also change the code to evals & exports checkpoints follow by epochs, not by step. Final, with the same size in the original repo, you should fine-tune instead of re-train from scratch, it will be ok.

[Mayur28]

Hi @voducman,

Thanks for the clarification. Considering that you are using LRS2 with image size = 192x192, are the images not very blurry? Resolution of the videos in LRS2 are 160x160 (and that is also before cropping the face which typically has a size around 110x 92). When the cropped out face is resized to be much larger than its original shape (110 x 92), would there not be a significant decrease in the clarity of the facial images? I've tried to finetune the model (when I wanted to verify that my own training pipeline works correctly), but I was hoping to atleast be able to successfully train the model to ensure that everything works, because thereafter, I am wanting to extend the model and consider alternate design choices (such as changing the type of ResNet block used, etc).

Thanks

[vskadandale]

Hi @Mayur28,

I am also facing the same problem. The loss is not getting below 0.68. I am using the original code from the repository with 96x96 images. Did you manage to fix this problem?

Thanks!

[Mayur28]

Hi @vskadandale,

Unfortunately for some reason, I was unable to get the loss below 0.68. Eventually, I decided to implement my own version from scratch whichh takes a while to converge, but the good news is that it converges (the lowest I have gotten is an eval loss of 0.3... I have not been able to get close to the 0.25 that is recommended in the Readme.md.

[zhanghm1995]

Hi, @Mayur28 ,

I also fine tuned the lip-sync expert network in my own dataset, although it's has converged to lower than ~0.25, but when I use this pretrained discrimator to train the Waw2Lip network, I can not get the good results. Indeed, I could get the a visually good-looking face images, but the lip motion was completely not synced with the audio.

So I wonder whether there are some metrics to evaluate the lip-sync expert performance rather than just look at the loss.

[sunwoo76]

I am training right now. This is my training logs ( I was modified the original repo):

Loss: 0.7035631441005935 - learning rate: 0.0001 - val loss: 0.6879662836299223 at epoch: 0: 100%|█| Loss: 0.6925660769976 - learning rate: 0.0001 - val loss: 0.69101863748887 at epoch: 1: 100%|█| 717/7 Loss: 0.6937094443845283 - learning rate: 0.0001 - val loss: 0.6934942813480601 at epoch: 2: 100%|█| Loss: 0.6964753221102173 - learning rate: 0.0001 - val loss: 0.6958373785018921 at epoch: 3: 100%|█| Loss: 0.6948814049591745 - learning rate: 0.0001 - val loss: 0.6927358402925379 at epoch: 4: 100%|█| Loss: 0.6957624335695245 - learning rate: 0.0001 - val loss: 0.6927358402925379 at epoch: 5: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000005.pth Loss: 0.6957362132923706 - learning rate: 0.0001 - val loss: 0.6922311712713802 at epoch: 5: 100%|█| Loss: 0.6941021396525235 - learning rate: 0.0001 - val loss: 0.6958037965437945 at epoch: 6: 100%|█| Loss: 0.6915607976946705 - learning rate: 0.0001 - val loss: 0.690644127481124 at epoch: 7: 100%|█| 7 Loss: 0.6906726746047058 - learning rate: 0.0001 - val loss: 0.6915773889597725 at epoch: 8: 100%|█| Loss: 0.690080951946051 - learning rate: 0.0001 - val loss: 0.6910958255038542 at epoch: 9: 100%|█| 7 Loss: 0.6887990220465474 - learning rate: 0.0001 - val loss: 0.6910958255038542 at epoch: 10: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000010.pth Loss: 0.688880747082842 - learning rate: 0.0001 - val loss: 0.6884867303511676 at epoch: 10: 100%|█| Loss: 0.6886253223093317 - learning rate: 0.0001 - val loss: 0.6820760369300842 at epoch: 11: 100%|█| Loss: 0.6794873835983968 - learning rate: 0.0001 - val loss: 0.6745607607504901 at epoch: 12: 100%|█| Loss: 0.6670064611913769 - learning rate: 0.0001 - val loss: 0.676604407675126 at epoch: 13: 100%|█| Loss: 0.6498011181045277 - learning rate: 0.0001 - val loss: 0.6408151738783893 at epoch: 14: 100%|█| Loss: 0.6376869129901491 - learning rate: 0.0001 - val loss: 0.6408151738783893 at epoch: 15: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000015.pth Loss: 0.6374346476718472 - learning rate: 0.0001 - val loss: 0.6302909886135775 at epoch: 15: 100%|█| Loss: 0.6209243929203252 - learning rate: 0.0001 - val loss: 0.5890622138977051 at epoch: 16: 100%|█| Loss: 0.5941242532001878 - learning rate: 0.0001 - val loss: 0.5624779112198773 at epoch: 17: 100%|█| Loss: 0.56167208396075 - learning rate: 0.0001 - val loss: 0.568172416266273 at epoch: 18: 100%|█| 71 Loss: 0.5466463566657722 - learning rate: 0.0001 - val loss: 0.5592151249156279 at epoch: 19: 100%|█| Loss: 0.5339792922721894 - learning rate: 0.0001 - val loss: 0.5592151249156279 at epoch: 20: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000020.pth Loss: 0.533931184305473 - learning rate: 0.0001 - val loss: 0.6127981701317955 at epoch: 20: 100%|█| Loss: 0.5249322705189055 - learning rate: 0.0001 - val loss: 0.5131637243663564 at epoch: 21: 100%|█| Loss: 0.5161000969436544 - learning rate: 0.0001 - val loss: 0.5203708129770616 at epoch: 22: 100%|█| Loss: 0.5090960334866949 - learning rate: 0.0001 - val loss: 0.48682357107891755 at epoch: 23: 100%|█ Loss: 0.5065931637499954 - learning rate: 0.0001 - val loss: 0.5037144615369684 at epoch: 24: 100%|█| Loss: 0.502112655041937 - learning rate: 0.0001 - val loss: 0.5037144615369684 at epoch: 25: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000025.pth Loss: 0.5021178523283789 - learning rate: 0.0001 - val loss: 0.485143698313657 at epoch: 25: 100%|█| Loss: 0.49395890506906653 - learning rate: 0.0001 - val loss: 0.5013865723329431 at epoch: 26: 100%|█ Loss: 0.4848401721052545 - learning rate: 0.0001 - val loss: 0.4662446151761448 at epoch: 27: 100%|█| Loss: 0.47855537859632047 - learning rate: 0.0001 - val loss: 0.46448684965862946 at epoch: 28: 100%| Loss: 0.47601954828080106 - learning rate: 0.0001 - val loss: 0.4636546881759868 at epoch: 29: 100%|█ Loss: 0.46970804585091896 - learning rate: 0.0001 - val loss: 0.4636546881759868 at epoch: 30: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000030.pth Loss: 0.4699182764755632 - learning rate: 0.0001 - val loss: 0.4619622212999007 at epoch: 30: 100%|█| Loss: 0.4648659408840674 - learning rate: 0.0001 - val loss: 0.4609289923134972 at epoch: 31: 100%|█| Loss: 0.45938822793328776 - learning rate: 0.0001 - val loss: 0.4740794248440686 at epoch: 32: 100%|█ Loss: 0.4599477379807535 - learning rate: 0.0001 - val loss: 0.4809425420620862 at epoch: 33: 100%|█| Loss: 0.45654312175188105 - learning rate: 0.0001 - val loss: 0.45326191537520466 at epoch: 34: 100%| Loss: 0.45342455105075624 - learning rate: 0.0001 - val loss: 0.45326191537520466 at epoch: 35: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000035.pth Loss: 0.45325286767472783 - learning rate: 0.0001 - val loss: 0.47276994235375347 at epoch: 35: 100%| Loss: 0.450026126627809 - learning rate: 0.0001 - val loss: 0.4686397366663989 at epoch: 36: 100%|█| Loss: 0.4464021891230819 - learning rate: 0.0001 - val loss: 0.45992140033665824 at epoch: 37: 100%|█ Loss: 0.44005519808419413 - learning rate: 0.0001 - val loss: 0.440639721996644 at epoch: 38: 100%|█| Loss: 0.4380961273338339 - learning rate: 0.0001 - val loss: 0.4316103195442873 at epoch: 39: 100%|█| Loss: 0.43915227126309325 - learning rate: 0.0001 - val loss: 0.4316103195442873 at epoch: 40: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000040.pth Loss: 0.43948496672398873 - learning rate: 0.0001 - val loss: 0.4332115492400001 at epoch: 40: 100%|█ Loss: 0.434869290852979 - learning rate: 0.0001 - val loss: 0.4537683076718274 at epoch: 41: 100%|█| Loss: 0.4278905222106678 - learning rate: 0.0001 - val loss: 0.41909335816607757 at epoch: 42: 100%|█ Loss: 0.43464335021946054 - learning rate: 0.0001 - val loss: 0.4331094114219441 at epoch: 43: 100%|█ Loss: 0.4268188434165891 - learning rate: 0.0001 - val loss: 0.4238515408600078 at epoch: 44: 100%|█| Loss: 0.4210699434969678 - learning rate: 0.0001 - val loss: 0.4238515408600078 at epoch: 45: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000045.pth Loss: 0.4211959919088223 - learning rate: 0.0001 - val loss: 0.42071713594829335 at epoch: 45: 100%|█ Loss: 0.4237418758736162 - learning rate: 0.0001 - val loss: 0.4431636684081134 at epoch: 46: 100%|█| Loss: 0.41622830285188045 - learning rate: 0.0001 - val loss: 0.4005160454441519 at epoch: 47: 100%|█ Loss: 0.4174851571083734 - learning rate: 0.0001 - val loss: 0.4179435389883378 at epoch: 48: 100%|█| Loss: 0.4172716616669792 - learning rate: 0.0001 - val loss: 0.4095070677645066 at epoch: 49: 100%|█| Loss: 0.41583427957482844 - learning rate: 0.0001 - val loss: 0.4095070677645066 at epoch: 50: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000050.pth Loss: 0.4156778309302017 - learning rate: 0.0001 - val loss: 0.41162291519782124 at epoch: 50: 100%|█ Loss: 0.41320431664731544 - learning rate: 0.0001 - val loss: 0.42958581798216877 at epoch: 51: 100%| Loss: 0.41230651324762957 - learning rate: 0.0001 - val loss: 0.38863546532743115 at epoch: 52: 100%| Loss: 0.4107196634866536 - learning rate: 0.0001 - val loss: 0.42489026925143075 at epoch: 53: 100%|█ Loss: 0.40585786544794483 - learning rate: 0.0001 - val loss: 0.3980054452138789 at epoch: 54: 100%|█ Loss: 0.4039612961464754 - learning rate: 0.0001 - val loss: 0.3980054452138789 at epoch: 55: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000055.pth Loss: 0.40400318470958885 - learning rate: 0.0001 - val loss: 0.3978138250463149 at epoch: 55: 100%|█ Loss: 0.4049618319819994 - learning rate: 0.0001 - val loss: 0.40917643378762636 at epoch: 56: 100%|█ Loss: 0.4007464385639507 - learning rate: 0.0001 - val loss: 0.38810308189953074 at epoch: 57: 100%|█ Loss: 0.3985109680866098 - learning rate: 0.0001 - val loss: 0.3736323661663953 at epoch: 58: 100%|█| Loss: 0.40083664098710364 - learning rate: 0.0001 - val loss: 0.40081219638095184 at epoch: 59: 100%| Loss: 0.3977955708790092 - learning rate: 0.0001 - val loss: 0.40081219638095184 at epoch: 60: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000060.pth Loss: 0.39780675425522827 - learning rate: 0.0001 - val loss: 0.4037880213821636 at epoch: 60: 100%|█ Loss: 0.3947395866469193 - learning rate: 0.0001 - val loss: 0.39472170612391305 at epoch: 61: 100%|█ Loss: 0.3960546641253361 - learning rate: 0.0001 - val loss: 0.3994171093491947 at epoch: 62: 100%|█| Loss: 0.3994179342248616 - learning rate: 0.0001 - val loss: 0.3968759091461406 at epoch: 63: 100%|█| Loss: 0.39311319232153263 - learning rate: 0.0001 - val loss: 0.4006752617218915 at epoch: 64: 100%|█ Loss: 0.387932076342612 - learning rate: 0.0001 - val loss: 0.4006752617218915 at epoch: 65: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000065.pth Loss: 0.38790877867942075 - learning rate: 0.0001 - val loss: 0.4133554434075075 at epoch: 65: 100%|█ Loss: 0.3932054582125971 - learning rate: 0.0001 - val loss: 0.3904358411536497 at epoch: 66: 100%|█| Loss: 0.38771996388375507 - learning rate: 0.0001 - val loss: 0.36515841413946715 at epoch: 67: 100%| Loss: 0.3914431752104473 - learning rate: 0.0001 - val loss: 0.4122841708800372 at epoch: 68: 100%|█| Loss: 0.3854125591834054 - learning rate: 0.0001 - val loss: 0.371323906323489 at epoch: 69: 100%|█| Loss: 0.3845594668937795 - learning rate: 0.0001 - val loss: 0.371323906323489 at epoch: 70: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000070.pth Loss: 0.38447392668517893 - learning rate: 0.0001 - val loss: 0.4026658114264993 at epoch: 70: 100%|█ Loss: 0.3848543026623533 - learning rate: 0.0001 - val loss: 0.3795255545307608 at epoch: 71: 100%|█| Loss: 0.3831122824753345 - learning rate: 0.0001 - val loss: 0.3858378108809976 at epoch: 72: 100%|█| Loss: 0.3818983071103089 - learning rate: 0.0001 - val loss: 0.39288316579426036 at epoch: 73: 100%|█ Loss: 0.3801079845079318 - learning rate: 0.0001 - val loss: 0.38747620933196125 at epoch: 74: 100%|█ Loss: 0.38191624851533157 - learning rate: 0.0001 - val loss: 0.38747620933196125 at epoch: 75: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000075.pth Loss: 0.3819815127420625 - learning rate: 0.0001 - val loss: 0.36888961055699515 at epoch: 75: 100%|█ Loss: 0.3792302764567704 - learning rate: 0.0001 - val loss: 0.363033264875412 at epoch: 76: 100%|█| Loss: 0.3787335364805272 - learning rate: 0.0001 - val loss: 0.38272398534943075 at epoch: 77: 100%|█ Loss: 0.3747828765311168 - learning rate: 0.0001 - val loss: 0.37849925721392913 at epoch: 78: 100%|█ Loss: 0.3727061307496772 - learning rate: 0.0001 - val loss: 0.3726185192080105 at epoch: 79: 100%|█| Loss: 0.3749956050315383 - learning rate: 0.0001 - val loss: 0.3726185192080105 at epoch: 80: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000080.pth Loss: 0.37504371609980425 - learning rate: 0.0001 - val loss: 0.37050309952567606 at epoch: 80: 100%| Loss: 0.3747331331298275 - learning rate: 0.0001 - val loss: 0.36530383895425234 at epoch: 81: 100%|█ Loss: 0.37707984173148246 - learning rate: 0.0001 - val loss: 0.35689589381217957 at epoch: 82: 100%| Loss: 0.3732697500677122 - learning rate: 0.0001 - val loss: 0.3590216654188493 at epoch: 83: 100%|█| Loss: 0.3694498129552711 - learning rate: 0.0001 - val loss: 0.36992430862258463 at epoch: 84: 100%|█ Loss: 0.37053039309948516 - learning rate: 0.0001 - val loss: 0.36992430862258463 at epoch: 85: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000085.pth Loss: 0.3704947035557057 - learning rate: 0.0001 - val loss: 0.38330086013850045 at epoch: 85: 100%|█ Loss: 0.36746956360223737 - learning rate: 0.0001 - val loss: 0.3864254951477051 at epoch: 86: 100%|█ Loss: 0.3703802171029473 - learning rate: 0.0001 - val loss: 0.3766837190179264 at epoch: 87: 100%|█| Loss: 0.36902527093388543 - learning rate: 0.0001 - val loss: 0.37262428332777586 at epoch: 88: 100%| Loss: 0.3665164634273641 - learning rate: 0.0001 - val loss: 0.3632035798886243 at epoch: 89: 100%|█| Loss: 0.3732048325948209 - learning rate: 0.0001 - val loss: 0.3632035798886243 at epoch: 90: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000090.pth Loss: 0.373401274283727 - learning rate: 0.0001 - val loss: 0.37565673624767976 at epoch: 90: 100%|█| Loss: 0.36413453232543735 - learning rate: 0.0001 - val loss: 0.36815528133336234 at epoch: 91: 100%| Loss: 0.36397053827135467 - learning rate: 0.0001 - val loss: 0.3481050168766695 at epoch: 92: 100%|█ Loss: 0.3620503853845463 - learning rate: 0.0001 - val loss: 0.3631353991873124 at epoch: 93: 100%|█| Loss: 0.3647868237776404 - learning rate: 0.0001 - val loss: 0.3595329803578994 at epoch: 94: 100%|█| Loss: 0.35972402996107855 - learning rate: 0.0001 - val loss: 0.3595329803578994 at epoch: 95: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000095.pth Loss: 0.3598175739019984 - learning rate: 0.0001 - val loss: 0.368411181604161 at epoch: 95: 100%|█| Loss: 0.3618277128853392 - learning rate: 0.0001 - val loss: 0.37604328113443714 at epoch: 96: 100%|█ Loss: 0.3625654221162969 - learning rate: 0.0001 - val loss: 0.3826728410580579 at epoch: 97: 100%|█| Loss: 0.359980691094445 - learning rate: 0.0001 - val loss: 0.35401028745314655 at epoch: 98: 100%|█| Loss: 0.3589852239879438 - learning rate: 0.0001 - val loss: 0.3632833396687227 at epoch: 99: 100%|█| Loss: 0.3591972673155742 - learning rate: 0.0001 - val loss: 0.3632833396687227 at epoch: 100: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000100.pth Loss: 0.3589290365098743 - learning rate: 0.0001 - val loss: 0.35630277149817524 at epoch: 100: 100%| Loss: 0.35614064462597234 - learning rate: 0.0001 - val loss: 0.3536679411635679 at epoch: 101: 100%| Loss: 0.35828276323557234 - learning rate: 0.0001 - val loss: 0.3508167740176706 at epoch: 102: 100%| Loss: 0.3540584946485078 - learning rate: 0.0001 - val loss: 0.35323924702756543 at epoch: 103: 100%| Loss: 0.35430986643086915 - learning rate: 0.0001 - val loss: 0.34429293870925903 at epoch: 104: 100% Loss: 0.3577814444108715 - learning rate: 0.0001 - val loss: 0.34429293870925903 at epoch: 105: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000105.pth Loss: 0.3578635099417994 - learning rate: 0.0001 - val loss: 0.3769519854994381 at epoch: 105: 100%|█ Loss: 0.3533500170898903 - learning rate: 0.0001 - val loss: 0.36215731326271505 at epoch: 106: 100%| Loss: 0.356513929583205 - learning rate: 0.0001 - val loss: 0.36290551809703603 at epoch: 107: 100%|█ Loss: 0.3555090062247493 - learning rate: 0.0001 - val loss: 0.36206039611031027 at epoch: 108: 100%| Loss: 0.35563033210266914 - learning rate: 0.0001 - val loss: 0.34595397114753723 at epoch: 109: 100% Loss: 0.35197701686004684 - learning rate: 0.0001 - val loss: 0.34595397114753723 at epoch: 110: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000110.pth Loss: 0.35189396344551316 - learning rate: 0.0001 - val loss: 0.3333049837280722 at epoch: 110: 100%| Loss: 0.35038518548178044 - learning rate: 0.0001 - val loss: 0.36820768959382 at epoch: 111: 100%|█| Loss: 0.35086454985198284 - learning rate: 0.0001 - val loss: 0.3493247926235199 at epoch: 112: 100%| Loss: 0.3508417098633581 - learning rate: 0.0001 - val loss: 0.3344658227527843 at epoch: 113: 100%|█ Loss: 0.3499371398193567 - learning rate: 0.0001 - val loss: 0.3559708349844989 at epoch: 114: 100%|█ Loss: 0.3488510699953114 - learning rate: 0.0001 - val loss: 0.3559708349844989 at epoch: 115: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000115.pth Loss: 0.34897961025873153 - learning rate: 0.0001 - val loss: 0.34428879969260273 at epoch: 115: 100% Loss: 0.34789785368671977 - learning rate: 0.0001 - val loss: 0.3504102580687579 at epoch: 116: 100%| Loss: 0.34687935713694185 - learning rate: 0.0001 - val loss: 0.34937093450742607 at epoch: 117: 100% Loss: 0.34958091423933807 - learning rate: 0.0001 - val loss: 0.35065576697097106 at epoch: 118: 100% Loss: 0.34731799226924465 - learning rate: 0.0001 - val loss: 0.3686371077509487 at epoch: 119: 100%| Loss: 0.35628663659012516 - learning rate: 0.0001 - val loss: 0.3686371077509487 at epoch: 120: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000120.pth Loss: 0.3565249024699423 - learning rate: 0.0001 - val loss: 0.3400667485068826 at epoch: 120: 100%|█ Loss: 0.3487435434860498 - learning rate: 0.0001 - val loss: 0.3453256154761595 at epoch: 121: 100%|█ Loss: 0.3515215716642981 - learning rate: 0.0001 - val loss: 0.32869693636894226 at epoch: 122: 100%| Loss: 0.35017006837008224 - learning rate: 0.0001 - val loss: 0.34082086997873645 at epoch: 123: 100% Loss: 0.34229803519684565 - learning rate: 0.0001 - val loss: 0.3332931294160731 at epoch: 124: 100%| Loss: 0.34396075596772757 - learning rate: 0.0001 - val loss: 0.3332931294160731 at epoch: 125: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000125.pth Loss: 0.3442547787058802 - learning rate: 0.0001 - val loss: 0.349171065232333 at epoch: 125: 100%|█| Loss: 0.34577042236404604 - learning rate: 0.0001 - val loss: 0.3645528414670159 at epoch: 126: 100%| Loss: 0.34132711608040783 - learning rate: 0.0001 - val loss: 0.3360985079232384 at epoch: 127: 100%| Loss: 0.3437456882648388 - learning rate: 0.0001 - val loss: 0.3382779079325059 at epoch: 128: 100%|█ Loss: 0.3492086415427167 - learning rate: 0.0001 - val loss: 0.34212653952486377 at epoch: 129: 100%| Loss: 0.3454139971450054 - learning rate: 0.0001 - val loss: 0.34212653952486377 at epoch: 130: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000130.pth Loss: 0.3456618153649893 - learning rate: 0.0001 - val loss: 0.3411855013931499 at epoch: 130: 100%|█ Loss: 0.34195975246456045 - learning rate: 0.0001 - val loss: 0.3484768604531008 at epoch: 131: 100%| Loss: 0.3413154762368156 - learning rate: 0.0001 - val loss: 0.3232783622601453 at epoch: 132: 100%|█ Loss: 0.34463959639291886 - learning rate: 0.0001 - val loss: 0.3466600670534022 at epoch: 133: 100%| Loss: 0.3417269390081594 - learning rate: 0.0001 - val loss: 0.3469894247896531 at epoch: 134: 100%|█ Loss: 0.34211664921578083 - learning rate: 0.0001 - val loss: 0.3469894247896531 at epoch: 135: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000135.pth Loss: 0.34235621625076945 - learning rate: 0.0001 - val loss: 0.32455766726942625 at epoch: 135: 100% Loss: 0.3417684130755096 - learning rate: 0.0001 - val loss: 0.34222139155163483 at epoch: 136: 100%| Loss: 0.34200943588711724 - learning rate: 0.0001 - val loss: 0.340353382860913 at epoch: 137: 100%|█ Loss: 0.3427103012176737 - learning rate: 0.0001 - val loss: 0.31848034700926614 at epoch: 138: 100%| Loss: 0.3389667235782456 - learning rate: 0.0001 - val loss: 0.33617665311869455 at epoch: 139: 100%| Loss: 0.3409421077861799 - learning rate: 0.0001 - val loss: 0.33617665311869455 at epoch: 140: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000140.pth Loss: 0.341106344279385 - learning rate: 0.0001 - val loss: 0.3356095131705789 at epoch: 140: 100%|█| Loss: 0.33813529564496364 - learning rate: 0.0001 - val loss: 0.3460811920025769 at epoch: 141: 100%| Loss: 0.33777565426989414 - learning rate: 0.0001 - val loss: 0.3389185053460738 at epoch: 142: 100%| Loss: 0.33634137785168705 - learning rate: 0.0001 - val loss: 0.33260248426128836 at epoch: 143: 100% Loss: 0.33574531278267733 - learning rate: 0.0001 - val loss: 0.3235555492779788 at epoch: 144: 100%| Loss: 0.33875305039090153 - learning rate: 0.0001 - val loss: 0.3235555492779788 at epoch: 145: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000145.pth Loss: 0.33885545482030305 - learning rate: 0.0001 - val loss: 0.35587654569569754 at epoch: 145: 100% Loss: 0.33802067140322517 - learning rate: 0.0001 - val loss: 0.33407504768932567 at epoch: 146: 100% Loss: 0.3392781635624262 - learning rate: 0.0001 - val loss: 0.3666040161076714 at epoch: 147: 100%|█ Loss: 0.3371390947653825 - learning rate: 0.0001 - val loss: 0.3289692428182153 at epoch: 148: 100%|█ Loss: 0.33789917689487026 - learning rate: 0.0001 - val loss: 0.3505098136032329 at epoch: 149: 100%| Loss: 0.3346090515940216 - learning rate: 0.0001 - val loss: 0.3505098136032329 at epoch: 150: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000150.pth Loss: 0.33460028960033106 - learning rate: 0.0001 - val loss: 0.33348899378496055 at epoch: 150: 100% Loss: 0.3352192599570568 - learning rate: 0.0001 - val loss: 0.34297194726326885 at epoch: 151: 100%| Loss: 0.33863768197668814 - learning rate: 0.0001 - val loss: 0.33424725953270407 at epoch: 152: 100% Loss: 0.33613811283680184 - learning rate: 0.0001 - val loss: 0.3319559430374819 at epoch: 153: 100%| Loss: 0.3353894404453879 - learning rate: 0.0001 - val loss: 0.33400768216918497 at epoch: 154: 100%| Loss: 0.33209147945296164 - learning rate: 0.0001 - val loss: 0.33400768216918497 at epoch: 155: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000155.pth Loss: 0.3320029157225557 - learning rate: 0.0001 - val loss: 0.34071629012332244 at epoch: 155: 100%| Loss: 0.33204671484905307 - learning rate: 0.0001 - val loss: 0.31705038161838756 at epoch: 156: 100% Loss: 0.33419463572558666 - learning rate: 0.0001 - val loss: 0.3177177625543931 at epoch: 157: 100%| Loss: 0.3301593000419968 - learning rate: 0.0001 - val loss: 0.3259129383984734 at epoch: 158: 100%|█ Loss: 0.3312155745476692 - learning rate: 0.0001 - val loss: 0.34573231549823985 at epoch: 159: 100%| Loss: 0.33380665605128146 - learning rate: 0.0001 - val loss: 0.34573231549823985 at epoch: 160: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000160.pth Loss: 0.33404608580025336 - learning rate: 0.0001 - val loss: 0.3204050204333137 at epoch: 160: 100%| Loss: 0.33061359949524266 - learning rate: 0.0001 - val loss: 0.3419087257455377 at epoch: 161: 100%| Loss: 0.3285083958003644 - learning rate: 0.0001 - val loss: 0.32915519440875335 at epoch: 162: 100%| Loss: 0.327556164568937 - learning rate: 0.0001 - val loss: 0.3519600980422076 at epoch: 163: 100%|█| Loss: 0.3347715123802049 - learning rate: 0.0001 - val loss: 0.3165829830309924 at epoch: 164: 100%|█ Loss: 0.33297615071664977 - learning rate: 0.0001 - val loss: 0.3165829830309924 at epoch: 165: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000165.pth Loss: 0.3331423436955262 - learning rate: 0.0001 - val loss: 0.3135868030435899 at epoch: 165: 100%|█ Loss: 0.33309593017380607 - learning rate: 0.0001 - val loss: 0.32883938270456653 at epoch: 166: 100% Loss: 0.33548486739854266 - learning rate: 0.0001 - val loss: 0.33917106074445386 at epoch: 167: 100% Loss: 0.3376981250453361 - learning rate: 0.0001 - val loss: 0.33148147779352527 at epoch: 168: 100%| Loss: 0.3325687631071029 - learning rate: 0.0001 - val loss: 0.33081743296454935 at epoch: 169: 100%| Loss: 0.3324487292358662 - learning rate: 0.0001 - val loss: 0.33081743296454935 at epoch: 170: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000170.pth Loss: 0.33248443889069257 - learning rate: 0.0001 - val loss: 0.3290778459871517 at epoch: 170: 100%| Loss: 0.32513701452287835 - learning rate: 0.0001 - val loss: 0.348345291965148 at epoch: 171: 100%|█ Loss: 0.32839437105499386 - learning rate: 0.0001 - val loss: 0.3147103365729837 at epoch: 172: 100%| Loss: 0.3292263769122513 - learning rate: 0.0001 - val loss: 0.3235371025169597 at epoch: 173: 100%|█ Loss: 0.33282532101312606 - learning rate: 0.0001 - val loss: 0.3237471466555315 at epoch: 174: 100%| Loss: 0.3292505520099368 - learning rate: 0.0001 - val loss: 0.3237471466555315 at epoch: 175: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000175.pth Loss: 0.32915210520206967 - learning rate: 0.0001 - val loss: 0.32510341616237864 at epoch: 175: 100% Loss: 0.32848219482054103 - learning rate: 0.0001 - val loss: 0.31360289016190696 at epoch: 176: 100% Loss: 0.32775541570562367 - learning rate: 0.0001 - val loss: 0.32836197754916024 at epoch: 177: 100% Loss: 0.32911148453274197 - learning rate: 0.0001 - val loss: 0.33613835538134856 at epoch: 178: 100% Loss: 0.3283213041359078 - learning rate: 0.0001 - val loss: 0.3272248716915355 at epoch: 179: 100%|█ Loss: 0.32704727372477177 - learning rate: 0.0001 - val loss: 0.3272248716915355 at epoch: 180: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000180.pth Loss: 0.3272566455760594 - learning rate: 0.0001 - val loss: 0.33723972124211926 at epoch: 180: 100%| Loss: 0.3257023029372283 - learning rate: 0.0001 - val loss: 0.32358309889540954 at epoch: 181: 100%| Loss: 0.3287189950961282 - learning rate: 0.0001 - val loss: 0.31701383871190686 at epoch: 182: 100%| Loss: 0.33806383744858964 - learning rate: 0.0001 - val loss: 0.33990464315694924 at epoch: 183: 100% Loss: 0.3448227711718119 - learning rate: 0.0001 - val loss: 0.323043777662165 at epoch: 184: 100%|█| Loss: 0.32528840119862024 - learning rate: 0.0001 - val loss: 0.323043777662165 at epoch: 185: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000185.pth Loss: 0.32527250814221725 - learning rate: 0.0001 - val loss: 0.31601231238421273 at epoch: 185: 100% Loss: 0.32251234111509897 - learning rate: 0.0001 - val loss: 0.30739615682293386 at epoch: 186: 100% Loss: 0.3262497771692409 - learning rate: 0.0001 - val loss: 0.3419104832060197 at epoch: 187: 100%|█ Loss: 0.3231987905385936 - learning rate: 0.0001 - val loss: 0.3456746129428639 at epoch: 188: 100%|█ Loss: 0.32693653987042576 - learning rate: 0.0001 - val loss: 0.32475994790301604 at epoch: 189: 100% Loss: 0.32213649243199627 - learning rate: 0.0001 - val loss: 0.32475994790301604 at epoch: 190: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000190.pth Loss: 0.32206250203453846 - learning rate: 0.0001 - val loss: 0.32578406614415784 at epoch: 190: 100% Loss: 0.33468802707630885 - learning rate: 0.0001 - val loss: 0.32397622045348673 at epoch: 191: 100% Loss: 0.32182675868148086 - learning rate: 0.0001 - val loss: 0.34383298193707185 at epoch: 192: 100% Loss: 0.32288886045561677 - learning rate: 0.0001 - val loss: 0.31198186909451203 at epoch: 193: 100% Loss: 0.3184754801927085 - learning rate: 0.0001 - val loss: 0.32010549569831176 at epoch: 194: 100%| Loss: 0.3190527330646968 - learning rate: 0.0001 - val loss: 0.32010549569831176 at epoch: 195: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000195.pth Loss: 0.3188870509714927 - learning rate: 0.0001 - val loss: 0.31590445953256946 at epoch: 195: 100%| Loss: 0.31914277071317704 - learning rate: 0.0001 - val loss: 0.32708724106059356 at epoch: 196: 100% Loss: 0.31704306400886306 - learning rate: 0.0001 - val loss: 0.31749794062446146 at epoch: 197: 100% Loss: 0.31728993576523984 - learning rate: 0.0001 - val loss: 0.3153178534087013 at epoch: 198: 100%| Loss: 0.32232077654601804 - learning rate: 0.0001 - val loss: 0.3280895422486698 at epoch: 199: 100%| Loss: 0.3228824969027295 - learning rate: 0.0001 - val loss: 0.3280895422486698 at epoch: 200: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000200.pth Loss: 0.32287279868724456 - learning rate: 0.0001 - val loss: 0.4071896163856282 at epoch: 200: 100%| Loss: 0.3160588775386039 - learning rate: 0.0001 - val loss: 0.30915284682722655 at epoch: 201: 100%| Loss: 0.3210019682491175 - learning rate: 0.0001 - val loss: 0.31598322356448455 at epoch: 202: 100%| Loss: 0.32000632737886625 - learning rate: 0.0001 - val loss: 0.31841721604852113 at epoch: 203: 100% Loss: 0.32569978021428175 - learning rate: 0.0001 - val loss: 0.30576730212744546 at epoch: 204: 100% Loss: 0.3216656012741547 - learning rate: 0.0001 - val loss: 0.30576730212744546 at epoch: 205: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000205.pth Loss: 0.32162326989978596 - learning rate: 0.0001 - val loss: 0.4041090800481684 at epoch: 205: 100%| Loss: 0.3229070930135134 - learning rate: 0.0001 - val loss: 0.31858688417602987 at epoch: 206: 100%| Loss: 0.3147415189147827 - learning rate: 0.0001 - val loss: 0.3235223661450779 at epoch: 207: 100%|█ Loss: 0.32002344245358655 - learning rate: 0.0001 - val loss: 0.3174469821593341 at epoch: 208: 100%| Loss: 0.3156113898488102 - learning rate: 0.0001 - val loss: 0.31989597047076507 at epoch: 209: 100%| Loss: 0.32095162807729655 - learning rate: 0.0001 - val loss: 0.31989597047076507 at epoch: 210: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000210.pth Loss: 0.32093586137770275 - learning rate: 0.0001 - val loss: 0.33956563122132244 at epoch: 210: 100% Loss: 0.3160227767219104 - learning rate: 0.0001 - val loss: 0.32169514719177694 at epoch: 211: 100%| Loss: 0.3175911024067359 - learning rate: 0.0001 - val loss: 0.3318972885608673 at epoch: 212: 100%|█ Loss: 0.31850152476229593 - learning rate: 0.0001 - val loss: 0.3349080576616175 at epoch: 213: 100%| Loss: 0.3189213678677212 - learning rate: 0.0001 - val loss: 0.3824808729045531 at epoch: 214: 100%|█ Loss: 0.32222099435729024 - learning rate: 0.0001 - val loss: 0.3824808729045531 at epoch: 215: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000215.pth Loss: 0.3225129536837547 - learning rate: 0.0001 - val loss: 0.32496700041434345 at epoch: 215: 100%| Loss: 0.3163468918166899 - learning rate: 0.0001 - val loss: 0.3155099679442013 at epoch: 216: 100%|█ Loss: 0.3181705278041805 - learning rate: 0.0001 - val loss: 0.2923184168689391 at epoch: 217: 100%|█ Loss: 0.31520228794512223 - learning rate: 0.0001 - val loss: 0.32073016902979684 at epoch: 218: 100% Loss: 0.31328052384168364 - learning rate: 0.0001 - val loss: 0.3084256368524888 at epoch: 219: 100%| Loss: 0.314738614016405 - learning rate: 0.0001 - val loss: 0.3084256368524888 at epoch: 220: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000220.pth Loss: 0.31483178794633393 - learning rate: 0.0001 - val loss: 0.4059099555015564 at epoch: 220: 100%| Loss: 0.3251883854551794 - learning rate: 0.0001 - val loss: 0.31899913181276884 at epoch: 221: 100%| Loss: 0.317881585746962 - learning rate: 0.0001 - val loss: 0.31094854719498577 at epoch: 222: 100%|█ Loss: 0.3171200509932583 - learning rate: 0.0001 - val loss: 0.3139798185404609 at epoch: 223: 100%|█ Loss: 0.3164741094100758 - learning rate: 0.0001 - val loss: 0.3251757253618801 at epoch: 224: 100%|█ Loss: 0.31757013108477244 - learning rate: 0.0001 - val loss: 0.3251757253618801 at epoch: 225: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000225.pth Loss: 0.31753265616617776 - learning rate: 0.0001 - val loss: 0.3073840264011832 at epoch: 225: 100%| Loss: 0.3136430283803155 - learning rate: 0.0001 - val loss: 0.32858657661606283 at epoch: 226: 100%| Loss: 0.31154451102558706 - learning rate: 0.0001 - val loss: 0.3088152750450022 at epoch: 227: 100%| Loss: 0.3164677088521681 - learning rate: 0.0001 - val loss: 0.331315336858525 at epoch: 228: 100%|█| Loss: 0.31428529164199215 - learning rate: 0.0001 - val loss: 0.3225905772517709 at epoch: 229: 100%| Loss: 0.3151915768088575 - learning rate: 0.0001 - val loss: 0.3225905772517709 at epoch: 230: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000230.pth Loss: 0.31526857360968863 - learning rate: 0.0001 - val loss: 0.3183743234942941 at epoch: 230: 100%| Loss: 0.3171861544771008 - learning rate: 0.0001 - val loss: 0.3250532816438114 at epoch: 231: 100%|█ Loss: 0.3126005499806863 - learning rate: 0.0001 - val loss: 0.31594983093878803 at epoch: 232: 100%| Loss: 0.31241105771214395 - learning rate: 0.0001 - val loss: 0.3324458371190464 at epoch: 233: 100%| Loss: 0.30969504739782633 - learning rate: 0.0001 - val loss: 0.3988265027018154 at epoch: 234: 100%| Loss: 0.3163540782667072 - learning rate: 0.0001 - val loss: 0.3988265027018154 at epoch: 235: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000235.pth Loss: 0.31643667399384484 - learning rate: 0.0001 - val loss: 0.3087838818045223 at epoch: 235: 100%| Loss: 0.31542253953559296 - learning rate: 0.0001 - val loss: 0.3142048400991103 at epoch: 236: 100%| Loss: 0.3157057684418879 - learning rate: 0.0001 - val loss: 0.32353735145400553 at epoch: 237: 100%| Loss: 0.3120209511520141 - learning rate: 0.0001 - val loss: 0.31309520847657146 at epoch: 238: 100%| Loss: 0.30913533819021044 - learning rate: 0.0001 - val loss: 0.3053393074694802 at epoch: 239: 100%| Loss: 0.3087995447135171 - learning rate: 0.0001 - val loss: 0.3053393074694802 at epoch: 240: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000240.pth Loss: 0.3087463332560937 - learning rate: 0.0001 - val loss: 0.3060222487239277 at epoch: 240: 100%|█ Loss: 0.31536922673467477 - learning rate: 0.0001 - val loss: 0.30904537176384644 at epoch: 241: 100% Loss: 0.31636830333053984 - learning rate: 0.0001 - val loss: 0.3328818792805952 at epoch: 242: 100%| Loss: 0.31180857620728064 - learning rate: 0.0001 - val loss: 0.32575685487074013 at epoch: 243: 100% Loss: 0.30974811510893424 - learning rate: 0.0001 - val loss: 0.309741834507269 at epoch: 244: 100%|█ Loss: 0.3107690839877342 - learning rate: 0.0001 - val loss: 0.309741834507269 at epoch: 245: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000245.pth Loss: 0.31067151745659205 - learning rate: 0.0001 - val loss: 0.3386328009998097 at epoch: 245: 100%| Loss: 0.3123398712281569 - learning rate: 0.0001 - val loss: 0.29813522801679726 at epoch: 246: 100%| Loss: 0.3130464750852545 - learning rate: 0.0001 - val loss: 0.31816019906717186 at epoch: 247: 100%| Loss: 0.3116480052221769 - learning rate: 0.0001 - val loss: 0.33110381224576163 at epoch: 248: 100%| Loss: 0.30647446732640765 - learning rate: 0.0001 - val loss: 0.4002171351629145 at epoch: 249: 100%| Loss: 0.30593346250789794 - learning rate: 0.0001 - val loss: 0.4002171351629145 at epoch: 250: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000250.pth Loss: 0.30605353545943065 - learning rate: 0.0001 - val loss: 0.3236594989019282 at epoch: 250: 100%| Loss: 0.31523266081364415 - learning rate: 0.0001 - val loss: 0.31214223188512463 at epoch: 251: 100% Loss: 0.3121627108239396 - learning rate: 0.0001 - val loss: 0.32304346386124105 at epoch: 252: 100%| Loss: 0.31201716769107385 - learning rate: 0.0001 - val loss: 0.40594975737964406 at epoch: 253: 100% Loss: 0.3132015399354272 - learning rate: 0.0001 - val loss: 0.29918953425744 at epoch: 254: 100%|█| Loss: 0.30866048826352177 - learning rate: 0.0001 - val loss: 0.29918953425744 at epoch: 255: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000255.pth Loss: 0.3088879721600308 - learning rate: 0.0001 - val loss: 0.3033159322598401 at epoch: 255: 100%|█ Loss: 0.30523368271491186 - learning rate: 0.0001 - val loss: 0.32445379215128284 at epoch: 256: 100% Loss: 0.3082592141304056 - learning rate: 0.0001 - val loss: 0.2987316995859146 at epoch: 257: 100%|█ Loss: 0.3077013487596392 - learning rate: 0.0001 - val loss: 0.32674457921701316 at epoch: 258: 100%| Loss: 0.31010557551098 - learning rate: 0.0001 - val loss: 0.3030623881255879 at epoch: 259: 100%|█| Loss: 0.3065339962239039 - learning rate: 0.0001 - val loss: 0.3030623881255879 at epoch: 260: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000260.pth Loss: 0.30648479753042696 - learning rate: 0.0001 - val loss: 0.2945685412953882 at epoch: 260: 100%| Loss: 0.3041540330465535 - learning rate: 0.0001 - val loss: 0.30476875252583446 at epoch: 261: 100%| Loss: 0.3068798369983915 - learning rate: 0.0001 - val loss: 0.31297055938664603 at epoch: 262: 100%| Loss: 0.30861558841029635 - learning rate: 0.0001 - val loss: 0.31001174186958985 at epoch: 263: 100% Loss: 0.31007959642752775 - learning rate: 0.0001 - val loss: 0.31483353060834546 at epoch: 264: 100% Loss: 0.307805921882391 - learning rate: 0.0001 - val loss: 0.31483353060834546 at epoch: 265: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000265.pth Loss: 0.3080047332218003 - learning rate: 0.0001 - val loss: 0.3080143954823999 at epoch: 265: 100%|█ Loss: 0.30392203399992057 - learning rate: 0.0001 - val loss: 0.29799437347580404 at epoch: 266: 100% Loss: 0.3053085238281819 - learning rate: 0.0001 - val loss: 0.3291468602769515 at epoch: 267: 100%|█ Loss: 0.30717476932240995 - learning rate: 5e-05 - val loss: 0.30195652737337 at epoch: 268: 100%|█| Loss: 0.3009409159214094 - learning rate: 5e-05 - val loss: 0.2819372073692434 at epoch: 269: 100%|█| Loss: 0.3001459057770628 - learning rate: 5e-05 - val loss: 0.2819372073692434 at epoch: 270: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000270.pth Loss: 0.3000148224074139 - learning rate: 5e-05 - val loss: 0.30267828615272746 at epoch: 270: 100%|█ Loss: 0.2992089938793222 - learning rate: 5e-05 - val loss: 0.30576135218143463 at epoch: 271: 100%|█ Loss: 0.297558757301155 - learning rate: 5e-05 - val loss: 0.31899135428316455 at epoch: 272: 100%|█| Loss: 0.297383855617861 - learning rate: 5e-05 - val loss: 0.30306536835782666 at epoch: 273: 100%|█| Loss: 0.2956093915503062 - learning rate: 5e-05 - val loss: 0.29790930537616506 at epoch: 274: 100%|█ Loss: 0.30059484900209493 - learning rate: 5e-05 - val loss: 0.29790930537616506 at epoch: 275: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000275.pth Loss: 0.30052305703964527 - learning rate: 5e-05 - val loss: 0.3061572313308716 at epoch: 275: 100%|█ Loss: 0.3003741754647578 - learning rate: 5e-05 - val loss: 0.29603279601125154 at epoch: 276: 100%|█ Loss: 0.299627613510048 - learning rate: 5e-05 - val loss: 0.28398248027352724 at epoch: 277: 100%|█| Loss: 0.3021080169301984 - learning rate: 5e-05 - val loss: 0.3111071569078109 at epoch: 278: 100%|█| Loss: 0.2939733659206906 - learning rate: 5e-05 - val loss: 0.2877907016698052 at epoch: 279: 100%|█| Loss: 0.298268929477677 - learning rate: 5e-05 - val loss: 0.2877907016698052 at epoch: 280: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000280.pth Loss: 0.2983553654188225 - learning rate: 5e-05 - val loss: 0.30654799675240235 at epoch: 280: 100%|█ Loss: 0.2983045185626468 - learning rate: 5e-05 - val loss: 0.3073072275694679 at epoch: 281: 100%|█| Loss: 0.2977428478052427 - learning rate: 5e-05 - val loss: 0.29910268678384666 at epoch: 282: 100%|█ Loss: 0.2954770494746031 - learning rate: 5e-05 - val loss: 0.3032520132906297 at epoch: 283: 100%|█| Loss: 0.2962064707919311 - learning rate: 5e-05 - val loss: 0.3157319461598116 at epoch: 284: 100%|█| Loss: 0.2965031956535811 - learning rate: 5e-05 - val loss: 0.3157319461598116 at epoch: 285: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000285.pth Loss: 0.29662289303133843 - learning rate: 5e-05 - val loss: 0.29873434760991263 at epoch: 285: 100%| Loss: 0.29265239134410104 - learning rate: 5e-05 - val loss: 0.32044096378719106 at epoch: 286: 100%| Loss: 0.29690624289266565 - learning rate: 5e-05 - val loss: 0.29064199416076436 at epoch: 287: 100%| Loss: 0.292572217319301 - learning rate: 5e-05 - val loss: 0.2932455557234147 at epoch: 288: 100%|█| Loss: 0.2910314104214539 - learning rate: 5e-05 - val loss: 0.3159342232872458 at epoch: 289: 100%|█| Loss: 0.2917812002104754 - learning rate: 5e-05 - val loss: 0.3159342232872458 at epoch: 290: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000290.pth Loss: 0.2919222831809205 - learning rate: 5e-05 - val loss: 0.3200300826745875 at epoch: 290: 100%|█| Loss: 0.2920384829554764 - learning rate: 5e-05 - val loss: 0.2998523282654145 at epoch: 291: 100%|█| Loss: 0.2966673892911009 - learning rate: 5e-05 - val loss: 0.2861257344484329 at epoch: 292: 100%|█| Loss: 0.29242345933884256 - learning rate: 5e-05 - val loss: 0.3010793976923999 at epoch: 293: 100%|█ Loss: 0.29116458153791175 - learning rate: 5e-05 - val loss: 0.29494749184917 at epoch: 294: 100%|█| Loss: 0.3043915748845932 - learning rate: 5e-05 - val loss: 0.29494749184917 at epoch: 295: 100%|▉| 7Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000295.pth Loss: 0.3045244020730714 - learning rate: 5e-05 - val loss: 0.29799241002868204 at epoch: 295: 100%|█ Loss: 0.29013394646481655 - learning rate: 5e-05 - val loss: 0.2721177050295998 at epoch: 296: 100%|█ Loss: 0.2957085180648367 - learning rate: 5e-05 - val loss: 0.2805605329134885 at epoch: 297: 100%|█| Loss: 0.29426202751602587 - learning rate: 5e-05 - val loss: 0.30168647012289834 at epoch: 298: 100%| Loss: 0.2920073376789253 - learning rate: 5e-05 - val loss: 0.2961472237811369 at epoch: 299: 100%|█| Loss: 0.2952373994087707 - learning rate: 5e-05 - val loss: 0.2961472237811369 at epoch: 300: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000300.pth Loss: 0.29522111856039596 - learning rate: 5e-05 - val loss: 0.28166491494459267 at epoch: 300: 100%| Loss: 0.29718377228145654 - learning rate: 5e-05 - val loss: 0.32492024057051716 at epoch: 301: 100%| Loss: 0.29203436547575967 - learning rate: 5e-05 - val loss: 0.29104062125963326 at epoch: 302: 100%| Loss: 0.29241493915829864 - learning rate: 5e-05 - val loss: 0.2915509176604888 at epoch: 303: 100%|█ Loss: 0.2910756430117156 - learning rate: 5e-05 - val loss: 0.3040669306236155 at epoch: 304: 100%|█| Loss: 0.2957818931665833 - learning rate: 5e-05 - val loss: 0.3040669306236155 at epoch: 305: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000305.pth Loss: 0.2959351596348455 - learning rate: 5e-05 - val loss: 0.3111670963904437 at epoch: 305: 100%|█| Loss: 0.29206380135843446 - learning rate: 5e-05 - val loss: 0.3021305343684028 at epoch: 306: 100%|█ Loss: 0.2906969288361455 - learning rate: 5e-05 - val loss: 0.30570460943614736 at epoch: 307: 100%|█ Loss: 0.2900724932447471 - learning rate: 5e-05 - val loss: 0.3324916222516228 at epoch: 308: 100%|█| Loss: 0.29348984774269654 - learning rate: 5e-05 - val loss: 0.3073321966563954 at epoch: 309: 100%|█ Loss: 0.2912182815622684 - learning rate: 5e-05 - val loss: 0.3073321966563954 at epoch: 310: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000310.pth Loss: 0.29140155960908326 - learning rate: 5e-05 - val loss: 0.3095227970796473 at epoch: 310: 100%|█ Loss: 0.28802871273900676 - learning rate: 5e-05 - val loss: 0.3084297224002726 at epoch: 311: 100%|█ Loss: 0.2929448010698688 - learning rate: 5e-05 - val loss: 0.3082868711036794 at epoch: 312: 100%|█

I am training right now. This is my training logs ( I was modified the original repo):

Loss: 0.7035631441005935 - learning rate: 0.0001 - val loss: 0.6879662836299223 at epoch: 0: 100%|█| Loss: 0.6925660769976 - learning rate: 0.0001 - val loss: 0.69101863748887 at epoch: 1: 100%|█| 717/7 Loss: 0.6937094443845283 - learning rate: 0.0001 - val loss: 0.6934942813480601 at epoch: 2: 100%|█| Loss: 0.6964753221102173 - learning rate: 0.0001 - val loss: 0.6958373785018921 at epoch: 3: 100%|█| Loss: 0.6948814049591745 - learning rate: 0.0001 - val loss: 0.6927358402925379 at epoch: 4: 100%|█| Loss: 0.6957624335695245 - learning rate: 0.0001 - val loss: 0.6927358402925379 at epoch: 5: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000005.pth Loss: 0.6957362132923706 - learning rate: 0.0001 - val loss: 0.6922311712713802 at epoch: 5: 100%|█| Loss: 0.6941021396525235 - learning rate: 0.0001 - val loss: 0.6958037965437945 at epoch: 6: 100%|█| Loss: 0.6915607976946705 - learning rate: 0.0001 - val loss: 0.690644127481124 at epoch: 7: 100%|█| 7 Loss: 0.6906726746047058 - learning rate: 0.0001 - val loss: 0.6915773889597725 at epoch: 8: 100%|█| Loss: 0.690080951946051 - learning rate: 0.0001 - val loss: 0.6910958255038542 at epoch: 9: 100%|█| 7 Loss: 0.6887990220465474 - learning rate: 0.0001 - val loss: 0.6910958255038542 at epoch: 10: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000010.pth Loss: 0.688880747082842 - learning rate: 0.0001 - val loss: 0.6884867303511676 at epoch: 10: 100%|█| Loss: 0.6886253223093317 - learning rate: 0.0001 - val loss: 0.6820760369300842 at epoch: 11: 100%|█| Loss: 0.6794873835983968 - learning rate: 0.0001 - val loss: 0.6745607607504901 at epoch: 12: 100%|█| Loss: 0.6670064611913769 - learning rate: 0.0001 - val loss: 0.676604407675126 at epoch: 13: 100%|█| Loss: 0.6498011181045277 - learning rate: 0.0001 - val loss: 0.6408151738783893 at epoch: 14: 100%|█| Loss: 0.6376869129901491 - learning rate: 0.0001 - val loss: 0.6408151738783893 at epoch: 15: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000015.pth Loss: 0.6374346476718472 - learning rate: 0.0001 - val loss: 0.6302909886135775 at epoch: 15: 100%|█| Loss: 0.6209243929203252 - learning rate: 0.0001 - val loss: 0.5890622138977051 at epoch: 16: 100%|█| Loss: 0.5941242532001878 - learning rate: 0.0001 - val loss: 0.5624779112198773 at epoch: 17: 100%|█| Loss: 0.56167208396075 - learning rate: 0.0001 - val loss: 0.568172416266273 at epoch: 18: 100%|█| 71 Loss: 0.5466463566657722 - learning rate: 0.0001 - val loss: 0.5592151249156279 at epoch: 19: 100%|█| Loss: 0.5339792922721894 - learning rate: 0.0001 - val loss: 0.5592151249156279 at epoch: 20: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000020.pth Loss: 0.533931184305473 - learning rate: 0.0001 - val loss: 0.6127981701317955 at epoch: 20: 100%|█| Loss: 0.5249322705189055 - learning rate: 0.0001 - val loss: 0.5131637243663564 at epoch: 21: 100%|█| Loss: 0.5161000969436544 - learning rate: 0.0001 - val loss: 0.5203708129770616 at epoch: 22: 100%|█| Loss: 0.5090960334866949 - learning rate: 0.0001 - val loss: 0.48682357107891755 at epoch: 23: 100%|█ Loss: 0.5065931637499954 - learning rate: 0.0001 - val loss: 0.5037144615369684 at epoch: 24: 100%|█| Loss: 0.502112655041937 - learning rate: 0.0001 - val loss: 0.5037144615369684 at epoch: 25: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000025.pth Loss: 0.5021178523283789 - learning rate: 0.0001 - val loss: 0.485143698313657 at epoch: 25: 100%|█| Loss: 0.49395890506906653 - learning rate: 0.0001 - val loss: 0.5013865723329431 at epoch: 26: 100%|█ Loss: 0.4848401721052545 - learning rate: 0.0001 - val loss: 0.4662446151761448 at epoch: 27: 100%|█| Loss: 0.47855537859632047 - learning rate: 0.0001 - val loss: 0.46448684965862946 at epoch: 28: 100%| Loss: 0.47601954828080106 - learning rate: 0.0001 - val loss: 0.4636546881759868 at epoch: 29: 100%|█ Loss: 0.46970804585091896 - learning rate: 0.0001 - val loss: 0.4636546881759868 at epoch: 30: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000030.pth Loss: 0.4699182764755632 - learning rate: 0.0001 - val loss: 0.4619622212999007 at epoch: 30: 100%|█| Loss: 0.4648659408840674 - learning rate: 0.0001 - val loss: 0.4609289923134972 at epoch: 31: 100%|█| Loss: 0.45938822793328776 - learning rate: 0.0001 - val loss: 0.4740794248440686 at epoch: 32: 100%|█ Loss: 0.4599477379807535 - learning rate: 0.0001 - val loss: 0.4809425420620862 at epoch: 33: 100%|█| Loss: 0.45654312175188105 - learning rate: 0.0001 - val loss: 0.45326191537520466 at epoch: 34: 100%| Loss: 0.45342455105075624 - learning rate: 0.0001 - val loss: 0.45326191537520466 at epoch: 35: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000035.pth Loss: 0.45325286767472783 - learning rate: 0.0001 - val loss: 0.47276994235375347 at epoch: 35: 100%| Loss: 0.450026126627809 - learning rate: 0.0001 - val loss: 0.4686397366663989 at epoch: 36: 100%|█| Loss: 0.4464021891230819 - learning rate: 0.0001 - val loss: 0.45992140033665824 at epoch: 37: 100%|█ Loss: 0.44005519808419413 - learning rate: 0.0001 - val loss: 0.440639721996644 at epoch: 38: 100%|█| Loss: 0.4380961273338339 - learning rate: 0.0001 - val loss: 0.4316103195442873 at epoch: 39: 100%|█| Loss: 0.43915227126309325 - learning rate: 0.0001 - val loss: 0.4316103195442873 at epoch: 40: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000040.pth Loss: 0.43948496672398873 - learning rate: 0.0001 - val loss: 0.4332115492400001 at epoch: 40: 100%|█ Loss: 0.434869290852979 - learning rate: 0.0001 - val loss: 0.4537683076718274 at epoch: 41: 100%|█| Loss: 0.4278905222106678 - learning rate: 0.0001 - val loss: 0.41909335816607757 at epoch: 42: 100%|█ Loss: 0.43464335021946054 - learning rate: 0.0001 - val loss: 0.4331094114219441 at epoch: 43: 100%|█ Loss: 0.4268188434165891 - learning rate: 0.0001 - val loss: 0.4238515408600078 at epoch: 44: 100%|█| Loss: 0.4210699434969678 - learning rate: 0.0001 - val loss: 0.4238515408600078 at epoch: 45: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000045.pth Loss: 0.4211959919088223 - learning rate: 0.0001 - val loss: 0.42071713594829335 at epoch: 45: 100%|█ Loss: 0.4237418758736162 - learning rate: 0.0001 - val loss: 0.4431636684081134 at epoch: 46: 100%|█| Loss: 0.41622830285188045 - learning rate: 0.0001 - val loss: 0.4005160454441519 at epoch: 47: 100%|█ Loss: 0.4174851571083734 - learning rate: 0.0001 - val loss: 0.4179435389883378 at epoch: 48: 100%|█| Loss: 0.4172716616669792 - learning rate: 0.0001 - val loss: 0.4095070677645066 at epoch: 49: 100%|█| Loss: 0.41583427957482844 - learning rate: 0.0001 - val loss: 0.4095070677645066 at epoch: 50: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000050.pth Loss: 0.4156778309302017 - learning rate: 0.0001 - val loss: 0.41162291519782124 at epoch: 50: 100%|█ Loss: 0.41320431664731544 - learning rate: 0.0001 - val loss: 0.42958581798216877 at epoch: 51: 100%| Loss: 0.41230651324762957 - learning rate: 0.0001 - val loss: 0.38863546532743115 at epoch: 52: 100%| Loss: 0.4107196634866536 - learning rate: 0.0001 - val loss: 0.42489026925143075 at epoch: 53: 100%|█ Loss: 0.40585786544794483 - learning rate: 0.0001 - val loss: 0.3980054452138789 at epoch: 54: 100%|█ Loss: 0.4039612961464754 - learning rate: 0.0001 - val loss: 0.3980054452138789 at epoch: 55: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000055.pth Loss: 0.40400318470958885 - learning rate: 0.0001 - val loss: 0.3978138250463149 at epoch: 55: 100%|█ Loss: 0.4049618319819994 - learning rate: 0.0001 - val loss: 0.40917643378762636 at epoch: 56: 100%|█ Loss: 0.4007464385639507 - learning rate: 0.0001 - val loss: 0.38810308189953074 at epoch: 57: 100%|█ Loss: 0.3985109680866098 - learning rate: 0.0001 - val loss: 0.3736323661663953 at epoch: 58: 100%|█| Loss: 0.40083664098710364 - learning rate: 0.0001 - val loss: 0.40081219638095184 at epoch: 59: 100%| Loss: 0.3977955708790092 - learning rate: 0.0001 - val loss: 0.40081219638095184 at epoch: 60: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000060.pth Loss: 0.39780675425522827 - learning rate: 0.0001 - val loss: 0.4037880213821636 at epoch: 60: 100%|█ Loss: 0.3947395866469193 - learning rate: 0.0001 - val loss: 0.39472170612391305 at epoch: 61: 100%|█ Loss: 0.3960546641253361 - learning rate: 0.0001 - val loss: 0.3994171093491947 at epoch: 62: 100%|█| Loss: 0.3994179342248616 - learning rate: 0.0001 - val loss: 0.3968759091461406 at epoch: 63: 100%|█| Loss: 0.39311319232153263 - learning rate: 0.0001 - val loss: 0.4006752617218915 at epoch: 64: 100%|█ Loss: 0.387932076342612 - learning rate: 0.0001 - val loss: 0.4006752617218915 at epoch: 65: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000065.pth Loss: 0.38790877867942075 - learning rate: 0.0001 - val loss: 0.4133554434075075 at epoch: 65: 100%|█ Loss: 0.3932054582125971 - learning rate: 0.0001 - val loss: 0.3904358411536497 at epoch: 66: 100%|█| Loss: 0.38771996388375507 - learning rate: 0.0001 - val loss: 0.36515841413946715 at epoch: 67: 100%| Loss: 0.3914431752104473 - learning rate: 0.0001 - val loss: 0.4122841708800372 at epoch: 68: 100%|█| Loss: 0.3854125591834054 - learning rate: 0.0001 - val loss: 0.371323906323489 at epoch: 69: 100%|█| Loss: 0.3845594668937795 - learning rate: 0.0001 - val loss: 0.371323906323489 at epoch: 70: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000070.pth Loss: 0.38447392668517893 - learning rate: 0.0001 - val loss: 0.4026658114264993 at epoch: 70: 100%|█ Loss: 0.3848543026623533 - learning rate: 0.0001 - val loss: 0.3795255545307608 at epoch: 71: 100%|█| Loss: 0.3831122824753345 - learning rate: 0.0001 - val loss: 0.3858378108809976 at epoch: 72: 100%|█| Loss: 0.3818983071103089 - learning rate: 0.0001 - val loss: 0.39288316579426036 at epoch: 73: 100%|█ Loss: 0.3801079845079318 - learning rate: 0.0001 - val loss: 0.38747620933196125 at epoch: 74: 100%|█ Loss: 0.38191624851533157 - learning rate: 0.0001 - val loss: 0.38747620933196125 at epoch: 75: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000075.pth Loss: 0.3819815127420625 - learning rate: 0.0001 - val loss: 0.36888961055699515 at epoch: 75: 100%|█ Loss: 0.3792302764567704 - learning rate: 0.0001 - val loss: 0.363033264875412 at epoch: 76: 100%|█| Loss: 0.3787335364805272 - learning rate: 0.0001 - val loss: 0.38272398534943075 at epoch: 77: 100%|█ Loss: 0.3747828765311168 - learning rate: 0.0001 - val loss: 0.37849925721392913 at epoch: 78: 100%|█ Loss: 0.3727061307496772 - learning rate: 0.0001 - val loss: 0.3726185192080105 at epoch: 79: 100%|█| Loss: 0.3749956050315383 - learning rate: 0.0001 - val loss: 0.3726185192080105 at epoch: 80: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000080.pth Loss: 0.37504371609980425 - learning rate: 0.0001 - val loss: 0.37050309952567606 at epoch: 80: 100%| Loss: 0.3747331331298275 - learning rate: 0.0001 - val loss: 0.36530383895425234 at epoch: 81: 100%|█ Loss: 0.37707984173148246 - learning rate: 0.0001 - val loss: 0.35689589381217957 at epoch: 82: 100%| Loss: 0.3732697500677122 - learning rate: 0.0001 - val loss: 0.3590216654188493 at epoch: 83: 100%|█| Loss: 0.3694498129552711 - learning rate: 0.0001 - val loss: 0.36992430862258463 at epoch: 84: 100%|█ Loss: 0.37053039309948516 - learning rate: 0.0001 - val loss: 0.36992430862258463 at epoch: 85: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000085.pth Loss: 0.3704947035557057 - learning rate: 0.0001 - val loss: 0.38330086013850045 at epoch: 85: 100%|█ Loss: 0.36746956360223737 - learning rate: 0.0001 - val loss: 0.3864254951477051 at epoch: 86: 100%|█ Loss: 0.3703802171029473 - learning rate: 0.0001 - val loss: 0.3766837190179264 at epoch: 87: 100%|█| Loss: 0.36902527093388543 - learning rate: 0.0001 - val loss: 0.37262428332777586 at epoch: 88: 100%| Loss: 0.3665164634273641 - learning rate: 0.0001 - val loss: 0.3632035798886243 at epoch: 89: 100%|█| Loss: 0.3732048325948209 - learning rate: 0.0001 - val loss: 0.3632035798886243 at epoch: 90: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000090.pth Loss: 0.373401274283727 - learning rate: 0.0001 - val loss: 0.37565673624767976 at epoch: 90: 100%|█| Loss: 0.36413453232543735 - learning rate: 0.0001 - val loss: 0.36815528133336234 at epoch: 91: 100%| Loss: 0.36397053827135467 - learning rate: 0.0001 - val loss: 0.3481050168766695 at epoch: 92: 100%|█ Loss: 0.3620503853845463 - learning rate: 0.0001 - val loss: 0.3631353991873124 at epoch: 93: 100%|█| Loss: 0.3647868237776404 - learning rate: 0.0001 - val loss: 0.3595329803578994 at epoch: 94: 100%|█| Loss: 0.35972402996107855 - learning rate: 0.0001 - val loss: 0.3595329803578994 at epoch: 95: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000095.pth Loss: 0.3598175739019984 - learning rate: 0.0001 - val loss: 0.368411181604161 at epoch: 95: 100%|█| Loss: 0.3618277128853392 - learning rate: 0.0001 - val loss: 0.37604328113443714 at epoch: 96: 100%|█ Loss: 0.3625654221162969 - learning rate: 0.0001 - val loss: 0.3826728410580579 at epoch: 97: 100%|█| Loss: 0.359980691094445 - learning rate: 0.0001 - val loss: 0.35401028745314655 at epoch: 98: 100%|█| Loss: 0.3589852239879438 - learning rate: 0.0001 - val loss: 0.3632833396687227 at epoch: 99: 100%|█| Loss: 0.3591972673155742 - learning rate: 0.0001 - val loss: 0.3632833396687227 at epoch: 100: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000100.pth Loss: 0.3589290365098743 - learning rate: 0.0001 - val loss: 0.35630277149817524 at epoch: 100: 100%| Loss: 0.35614064462597234 - learning rate: 0.0001 - val loss: 0.3536679411635679 at epoch: 101: 100%| Loss: 0.35828276323557234 - learning rate: 0.0001 - val loss: 0.3508167740176706 at epoch: 102: 100%| Loss: 0.3540584946485078 - learning rate: 0.0001 - val loss: 0.35323924702756543 at epoch: 103: 100%| Loss: 0.35430986643086915 - learning rate: 0.0001 - val loss: 0.34429293870925903 at epoch: 104: 100% Loss: 0.3577814444108715 - learning rate: 0.0001 - val loss: 0.34429293870925903 at epoch: 105: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000105.pth Loss: 0.3578635099417994 - learning rate: 0.0001 - val loss: 0.3769519854994381 at epoch: 105: 100%|█ Loss: 0.3533500170898903 - learning rate: 0.0001 - val loss: 0.36215731326271505 at epoch: 106: 100%| Loss: 0.356513929583205 - learning rate: 0.0001 - val loss: 0.36290551809703603 at epoch: 107: 100%|█ Loss: 0.3555090062247493 - learning rate: 0.0001 - val loss: 0.36206039611031027 at epoch: 108: 100%| Loss: 0.35563033210266914 - learning rate: 0.0001 - val loss: 0.34595397114753723 at epoch: 109: 100% Loss: 0.35197701686004684 - learning rate: 0.0001 - val loss: 0.34595397114753723 at epoch: 110: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000110.pth Loss: 0.35189396344551316 - learning rate: 0.0001 - val loss: 0.3333049837280722 at epoch: 110: 100%| Loss: 0.35038518548178044 - learning rate: 0.0001 - val loss: 0.36820768959382 at epoch: 111: 100%|█| Loss: 0.35086454985198284 - learning rate: 0.0001 - val loss: 0.3493247926235199 at epoch: 112: 100%| Loss: 0.3508417098633581 - learning rate: 0.0001 - val loss: 0.3344658227527843 at epoch: 113: 100%|█ Loss: 0.3499371398193567 - learning rate: 0.0001 - val loss: 0.3559708349844989 at epoch: 114: 100%|█ Loss: 0.3488510699953114 - learning rate: 0.0001 - val loss: 0.3559708349844989 at epoch: 115: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000115.pth Loss: 0.34897961025873153 - learning rate: 0.0001 - val loss: 0.34428879969260273 at epoch: 115: 100% Loss: 0.34789785368671977 - learning rate: 0.0001 - val loss: 0.3504102580687579 at epoch: 116: 100%| Loss: 0.34687935713694185 - learning rate: 0.0001 - val loss: 0.34937093450742607 at epoch: 117: 100% Loss: 0.34958091423933807 - learning rate: 0.0001 - val loss: 0.35065576697097106 at epoch: 118: 100% Loss: 0.34731799226924465 - learning rate: 0.0001 - val loss: 0.3686371077509487 at epoch: 119: 100%| Loss: 0.35628663659012516 - learning rate: 0.0001 - val loss: 0.3686371077509487 at epoch: 120: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000120.pth Loss: 0.3565249024699423 - learning rate: 0.0001 - val loss: 0.3400667485068826 at epoch: 120: 100%|█ Loss: 0.3487435434860498 - learning rate: 0.0001 - val loss: 0.3453256154761595 at epoch: 121: 100%|█ Loss: 0.3515215716642981 - learning rate: 0.0001 - val loss: 0.32869693636894226 at epoch: 122: 100%| Loss: 0.35017006837008224 - learning rate: 0.0001 - val loss: 0.34082086997873645 at epoch: 123: 100% Loss: 0.34229803519684565 - learning rate: 0.0001 - val loss: 0.3332931294160731 at epoch: 124: 100%| Loss: 0.34396075596772757 - learning rate: 0.0001 - val loss: 0.3332931294160731 at epoch: 125: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000125.pth Loss: 0.3442547787058802 - learning rate: 0.0001 - val loss: 0.349171065232333 at epoch: 125: 100%|█| Loss: 0.34577042236404604 - learning rate: 0.0001 - val loss: 0.3645528414670159 at epoch: 126: 100%| Loss: 0.34132711608040783 - learning rate: 0.0001 - val loss: 0.3360985079232384 at epoch: 127: 100%| Loss: 0.3437456882648388 - learning rate: 0.0001 - val loss: 0.3382779079325059 at epoch: 128: 100%|█ Loss: 0.3492086415427167 - learning rate: 0.0001 - val loss: 0.34212653952486377 at epoch: 129: 100%| Loss: 0.3454139971450054 - learning rate: 0.0001 - val loss: 0.34212653952486377 at epoch: 130: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000130.pth Loss: 0.3456618153649893 - learning rate: 0.0001 - val loss: 0.3411855013931499 at epoch: 130: 100%|█ Loss: 0.34195975246456045 - learning rate: 0.0001 - val loss: 0.3484768604531008 at epoch: 131: 100%| Loss: 0.3413154762368156 - learning rate: 0.0001 - val loss: 0.3232783622601453 at epoch: 132: 100%|█ Loss: 0.34463959639291886 - learning rate: 0.0001 - val loss: 0.3466600670534022 at epoch: 133: 100%| Loss: 0.3417269390081594 - learning rate: 0.0001 - val loss: 0.3469894247896531 at epoch: 134: 100%|█ Loss: 0.34211664921578083 - learning rate: 0.0001 - val loss: 0.3469894247896531 at epoch: 135: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000135.pth Loss: 0.34235621625076945 - learning rate: 0.0001 - val loss: 0.32455766726942625 at epoch: 135: 100% Loss: 0.3417684130755096 - learning rate: 0.0001 - val loss: 0.34222139155163483 at epoch: 136: 100%| Loss: 0.34200943588711724 - learning rate: 0.0001 - val loss: 0.340353382860913 at epoch: 137: 100%|█ Loss: 0.3427103012176737 - learning rate: 0.0001 - val loss: 0.31848034700926614 at epoch: 138: 100%| Loss: 0.3389667235782456 - learning rate: 0.0001 - val loss: 0.33617665311869455 at epoch: 139: 100%| Loss: 0.3409421077861799 - learning rate: 0.0001 - val loss: 0.33617665311869455 at epoch: 140: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000140.pth Loss: 0.341106344279385 - learning rate: 0.0001 - val loss: 0.3356095131705789 at epoch: 140: 100%|█| Loss: 0.33813529564496364 - learning rate: 0.0001 - val loss: 0.3460811920025769 at epoch: 141: 100%| Loss: 0.33777565426989414 - learning rate: 0.0001 - val loss: 0.3389185053460738 at epoch: 142: 100%| Loss: 0.33634137785168705 - learning rate: 0.0001 - val loss: 0.33260248426128836 at epoch: 143: 100% Loss: 0.33574531278267733 - learning rate: 0.0001 - val loss: 0.3235555492779788 at epoch: 144: 100%| Loss: 0.33875305039090153 - learning rate: 0.0001 - val loss: 0.3235555492779788 at epoch: 145: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000145.pth Loss: 0.33885545482030305 - learning rate: 0.0001 - val loss: 0.35587654569569754 at epoch: 145: 100% Loss: 0.33802067140322517 - learning rate: 0.0001 - val loss: 0.33407504768932567 at epoch: 146: 100% Loss: 0.3392781635624262 - learning rate: 0.0001 - val loss: 0.3666040161076714 at epoch: 147: 100%|█ Loss: 0.3371390947653825 - learning rate: 0.0001 - val loss: 0.3289692428182153 at epoch: 148: 100%|█ Loss: 0.33789917689487026 - learning rate: 0.0001 - val loss: 0.3505098136032329 at epoch: 149: 100%| Loss: 0.3346090515940216 - learning rate: 0.0001 - val loss: 0.3505098136032329 at epoch: 150: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000150.pth Loss: 0.33460028960033106 - learning rate: 0.0001 - val loss: 0.33348899378496055 at epoch: 150: 100% Loss: 0.3352192599570568 - learning rate: 0.0001 - val loss: 0.34297194726326885 at epoch: 151: 100%| Loss: 0.33863768197668814 - learning rate: 0.0001 - val loss: 0.33424725953270407 at epoch: 152: 100% Loss: 0.33613811283680184 - learning rate: 0.0001 - val loss: 0.3319559430374819 at epoch: 153: 100%| Loss: 0.3353894404453879 - learning rate: 0.0001 - val loss: 0.33400768216918497 at epoch: 154: 100%| Loss: 0.33209147945296164 - learning rate: 0.0001 - val loss: 0.33400768216918497 at epoch: 155: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000155.pth Loss: 0.3320029157225557 - learning rate: 0.0001 - val loss: 0.34071629012332244 at epoch: 155: 100%| Loss: 0.33204671484905307 - learning rate: 0.0001 - val loss: 0.31705038161838756 at epoch: 156: 100% Loss: 0.33419463572558666 - learning rate: 0.0001 - val loss: 0.3177177625543931 at epoch: 157: 100%| Loss: 0.3301593000419968 - learning rate: 0.0001 - val loss: 0.3259129383984734 at epoch: 158: 100%|█ Loss: 0.3312155745476692 - learning rate: 0.0001 - val loss: 0.34573231549823985 at epoch: 159: 100%| Loss: 0.33380665605128146 - learning rate: 0.0001 - val loss: 0.34573231549823985 at epoch: 160: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000160.pth Loss: 0.33404608580025336 - learning rate: 0.0001 - val loss: 0.3204050204333137 at epoch: 160: 100%| Loss: 0.33061359949524266 - learning rate: 0.0001 - val loss: 0.3419087257455377 at epoch: 161: 100%| Loss: 0.3285083958003644 - learning rate: 0.0001 - val loss: 0.32915519440875335 at epoch: 162: 100%| Loss: 0.327556164568937 - learning rate: 0.0001 - val loss: 0.3519600980422076 at epoch: 163: 100%|█| Loss: 0.3347715123802049 - learning rate: 0.0001 - val loss: 0.3165829830309924 at epoch: 164: 100%|█ Loss: 0.33297615071664977 - learning rate: 0.0001 - val loss: 0.3165829830309924 at epoch: 165: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000165.pth Loss: 0.3331423436955262 - learning rate: 0.0001 - val loss: 0.3135868030435899 at epoch: 165: 100%|█ Loss: 0.33309593017380607 - learning rate: 0.0001 - val loss: 0.32883938270456653 at epoch: 166: 100% Loss: 0.33548486739854266 - learning rate: 0.0001 - val loss: 0.33917106074445386 at epoch: 167: 100% Loss: 0.3376981250453361 - learning rate: 0.0001 - val loss: 0.33148147779352527 at epoch: 168: 100%| Loss: 0.3325687631071029 - learning rate: 0.0001 - val loss: 0.33081743296454935 at epoch: 169: 100%| Loss: 0.3324487292358662 - learning rate: 0.0001 - val loss: 0.33081743296454935 at epoch: 170: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000170.pth Loss: 0.33248443889069257 - learning rate: 0.0001 - val loss: 0.3290778459871517 at epoch: 170: 100%| Loss: 0.32513701452287835 - learning rate: 0.0001 - val loss: 0.348345291965148 at epoch: 171: 100%|█ Loss: 0.32839437105499386 - learning rate: 0.0001 - val loss: 0.3147103365729837 at epoch: 172: 100%| Loss: 0.3292263769122513 - learning rate: 0.0001 - val loss: 0.3235371025169597 at epoch: 173: 100%|█ Loss: 0.33282532101312606 - learning rate: 0.0001 - val loss: 0.3237471466555315 at epoch: 174: 100%| Loss: 0.3292505520099368 - learning rate: 0.0001 - val loss: 0.3237471466555315 at epoch: 175: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000175.pth Loss: 0.32915210520206967 - learning rate: 0.0001 - val loss: 0.32510341616237864 at epoch: 175: 100% Loss: 0.32848219482054103 - learning rate: 0.0001 - val loss: 0.31360289016190696 at epoch: 176: 100% Loss: 0.32775541570562367 - learning rate: 0.0001 - val loss: 0.32836197754916024 at epoch: 177: 100% Loss: 0.32911148453274197 - learning rate: 0.0001 - val loss: 0.33613835538134856 at epoch: 178: 100% Loss: 0.3283213041359078 - learning rate: 0.0001 - val loss: 0.3272248716915355 at epoch: 179: 100%|█ Loss: 0.32704727372477177 - learning rate: 0.0001 - val loss: 0.3272248716915355 at epoch: 180: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000180.pth Loss: 0.3272566455760594 - learning rate: 0.0001 - val loss: 0.33723972124211926 at epoch: 180: 100%| Loss: 0.3257023029372283 - learning rate: 0.0001 - val loss: 0.32358309889540954 at epoch: 181: 100%| Loss: 0.3287189950961282 - learning rate: 0.0001 - val loss: 0.31701383871190686 at epoch: 182: 100%| Loss: 0.33806383744858964 - learning rate: 0.0001 - val loss: 0.33990464315694924 at epoch: 183: 100% Loss: 0.3448227711718119 - learning rate: 0.0001 - val loss: 0.323043777662165 at epoch: 184: 100%|█| Loss: 0.32528840119862024 - learning rate: 0.0001 - val loss: 0.323043777662165 at epoch: 185: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000185.pth Loss: 0.32527250814221725 - learning rate: 0.0001 - val loss: 0.31601231238421273 at epoch: 185: 100% Loss: 0.32251234111509897 - learning rate: 0.0001 - val loss: 0.30739615682293386 at epoch: 186: 100% Loss: 0.3262497771692409 - learning rate: 0.0001 - val loss: 0.3419104832060197 at epoch: 187: 100%|█ Loss: 0.3231987905385936 - learning rate: 0.0001 - val loss: 0.3456746129428639 at epoch: 188: 100%|█ Loss: 0.32693653987042576 - learning rate: 0.0001 - val loss: 0.32475994790301604 at epoch: 189: 100% Loss: 0.32213649243199627 - learning rate: 0.0001 - val loss: 0.32475994790301604 at epoch: 190: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000190.pth Loss: 0.32206250203453846 - learning rate: 0.0001 - val loss: 0.32578406614415784 at epoch: 190: 100% Loss: 0.33468802707630885 - learning rate: 0.0001 - val loss: 0.32397622045348673 at epoch: 191: 100% Loss: 0.32182675868148086 - learning rate: 0.0001 - val loss: 0.34383298193707185 at epoch: 192: 100% Loss: 0.32288886045561677 - learning rate: 0.0001 - val loss: 0.31198186909451203 at epoch: 193: 100% Loss: 0.3184754801927085 - learning rate: 0.0001 - val loss: 0.32010549569831176 at epoch: 194: 100%| Loss: 0.3190527330646968 - learning rate: 0.0001 - val loss: 0.32010549569831176 at epoch: 195: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000195.pth Loss: 0.3188870509714927 - learning rate: 0.0001 - val loss: 0.31590445953256946 at epoch: 195: 100%| Loss: 0.31914277071317704 - learning rate: 0.0001 - val loss: 0.32708724106059356 at epoch: 196: 100% Loss: 0.31704306400886306 - learning rate: 0.0001 - val loss: 0.31749794062446146 at epoch: 197: 100% Loss: 0.31728993576523984 - learning rate: 0.0001 - val loss: 0.3153178534087013 at epoch: 198: 100%| Loss: 0.32232077654601804 - learning rate: 0.0001 - val loss: 0.3280895422486698 at epoch: 199: 100%| Loss: 0.3228824969027295 - learning rate: 0.0001 - val loss: 0.3280895422486698 at epoch: 200: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000200.pth Loss: 0.32287279868724456 - learning rate: 0.0001 - val loss: 0.4071896163856282 at epoch: 200: 100%| Loss: 0.3160588775386039 - learning rate: 0.0001 - val loss: 0.30915284682722655 at epoch: 201: 100%| Loss: 0.3210019682491175 - learning rate: 0.0001 - val loss: 0.31598322356448455 at epoch: 202: 100%| Loss: 0.32000632737886625 - learning rate: 0.0001 - val loss: 0.31841721604852113 at epoch: 203: 100% Loss: 0.32569978021428175 - learning rate: 0.0001 - val loss: 0.30576730212744546 at epoch: 204: 100% Loss: 0.3216656012741547 - learning rate: 0.0001 - val loss: 0.30576730212744546 at epoch: 205: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000205.pth Loss: 0.32162326989978596 - learning rate: 0.0001 - val loss: 0.4041090800481684 at epoch: 205: 100%| Loss: 0.3229070930135134 - learning rate: 0.0001 - val loss: 0.31858688417602987 at epoch: 206: 100%| Loss: 0.3147415189147827 - learning rate: 0.0001 - val loss: 0.3235223661450779 at epoch: 207: 100%|█ Loss: 0.32002344245358655 - learning rate: 0.0001 - val loss: 0.3174469821593341 at epoch: 208: 100%| Loss: 0.3156113898488102 - learning rate: 0.0001 - val loss: 0.31989597047076507 at epoch: 209: 100%| Loss: 0.32095162807729655 - learning rate: 0.0001 - val loss: 0.31989597047076507 at epoch: 210: 100%Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000210.pth Loss: 0.32093586137770275 - learning rate: 0.0001 - val loss: 0.33956563122132244 at epoch: 210: 100% Loss: 0.3160227767219104 - learning rate: 0.0001 - val loss: 0.32169514719177694 at epoch: 211: 100%| Loss: 0.3175911024067359 - learning rate: 0.0001 - val loss: 0.3318972885608673 at epoch: 212: 100%|█ Loss: 0.31850152476229593 - learning rate: 0.0001 - val loss: 0.3349080576616175 at epoch: 213: 100%| Loss: 0.3189213678677212 - learning rate: 0.0001 - val loss: 0.3824808729045531 at epoch: 214: 100%|█ Loss: 0.32222099435729024 - learning rate: 0.0001 - val loss: 0.3824808729045531 at epoch: 215: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000215.pth Loss: 0.3225129536837547 - learning rate: 0.0001 - val loss: 0.32496700041434345 at epoch: 215: 100%| Loss: 0.3163468918166899 - learning rate: 0.0001 - val loss: 0.3155099679442013 at epoch: 216: 100%|█ Loss: 0.3181705278041805 - learning rate: 0.0001 - val loss: 0.2923184168689391 at epoch: 217: 100%|█ Loss: 0.31520228794512223 - learning rate: 0.0001 - val loss: 0.32073016902979684 at epoch: 218: 100% Loss: 0.31328052384168364 - learning rate: 0.0001 - val loss: 0.3084256368524888 at epoch: 219: 100%| Loss: 0.314738614016405 - learning rate: 0.0001 - val loss: 0.3084256368524888 at epoch: 220: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000220.pth Loss: 0.31483178794633393 - learning rate: 0.0001 - val loss: 0.4059099555015564 at epoch: 220: 100%| Loss: 0.3251883854551794 - learning rate: 0.0001 - val loss: 0.31899913181276884 at epoch: 221: 100%| Loss: 0.317881585746962 - learning rate: 0.0001 - val loss: 0.31094854719498577 at epoch: 222: 100%|█ Loss: 0.3171200509932583 - learning rate: 0.0001 - val loss: 0.3139798185404609 at epoch: 223: 100%|█ Loss: 0.3164741094100758 - learning rate: 0.0001 - val loss: 0.3251757253618801 at epoch: 224: 100%|█ Loss: 0.31757013108477244 - learning rate: 0.0001 - val loss: 0.3251757253618801 at epoch: 225: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000225.pth Loss: 0.31753265616617776 - learning rate: 0.0001 - val loss: 0.3073840264011832 at epoch: 225: 100%| Loss: 0.3136430283803155 - learning rate: 0.0001 - val loss: 0.32858657661606283 at epoch: 226: 100%| Loss: 0.31154451102558706 - learning rate: 0.0001 - val loss: 0.3088152750450022 at epoch: 227: 100%| Loss: 0.3164677088521681 - learning rate: 0.0001 - val loss: 0.331315336858525 at epoch: 228: 100%|█| Loss: 0.31428529164199215 - learning rate: 0.0001 - val loss: 0.3225905772517709 at epoch: 229: 100%| Loss: 0.3151915768088575 - learning rate: 0.0001 - val loss: 0.3225905772517709 at epoch: 230: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000230.pth Loss: 0.31526857360968863 - learning rate: 0.0001 - val loss: 0.3183743234942941 at epoch: 230: 100%| Loss: 0.3171861544771008 - learning rate: 0.0001 - val loss: 0.3250532816438114 at epoch: 231: 100%|█ Loss: 0.3126005499806863 - learning rate: 0.0001 - val loss: 0.31594983093878803 at epoch: 232: 100%| Loss: 0.31241105771214395 - learning rate: 0.0001 - val loss: 0.3324458371190464 at epoch: 233: 100%| Loss: 0.30969504739782633 - learning rate: 0.0001 - val loss: 0.3988265027018154 at epoch: 234: 100%| Loss: 0.3163540782667072 - learning rate: 0.0001 - val loss: 0.3988265027018154 at epoch: 235: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000235.pth Loss: 0.31643667399384484 - learning rate: 0.0001 - val loss: 0.3087838818045223 at epoch: 235: 100%| Loss: 0.31542253953559296 - learning rate: 0.0001 - val loss: 0.3142048400991103 at epoch: 236: 100%| Loss: 0.3157057684418879 - learning rate: 0.0001 - val loss: 0.32353735145400553 at epoch: 237: 100%| Loss: 0.3120209511520141 - learning rate: 0.0001 - val loss: 0.31309520847657146 at epoch: 238: 100%| Loss: 0.30913533819021044 - learning rate: 0.0001 - val loss: 0.3053393074694802 at epoch: 239: 100%| Loss: 0.3087995447135171 - learning rate: 0.0001 - val loss: 0.3053393074694802 at epoch: 240: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000240.pth Loss: 0.3087463332560937 - learning rate: 0.0001 - val loss: 0.3060222487239277 at epoch: 240: 100%|█ Loss: 0.31536922673467477 - learning rate: 0.0001 - val loss: 0.30904537176384644 at epoch: 241: 100% Loss: 0.31636830333053984 - learning rate: 0.0001 - val loss: 0.3328818792805952 at epoch: 242: 100%| Loss: 0.31180857620728064 - learning rate: 0.0001 - val loss: 0.32575685487074013 at epoch: 243: 100% Loss: 0.30974811510893424 - learning rate: 0.0001 - val loss: 0.309741834507269 at epoch: 244: 100%|█ Loss: 0.3107690839877342 - learning rate: 0.0001 - val loss: 0.309741834507269 at epoch: 245: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000245.pth Loss: 0.31067151745659205 - learning rate: 0.0001 - val loss: 0.3386328009998097 at epoch: 245: 100%| Loss: 0.3123398712281569 - learning rate: 0.0001 - val loss: 0.29813522801679726 at epoch: 246: 100%| Loss: 0.3130464750852545 - learning rate: 0.0001 - val loss: 0.31816019906717186 at epoch: 247: 100%| Loss: 0.3116480052221769 - learning rate: 0.0001 - val loss: 0.33110381224576163 at epoch: 248: 100%| Loss: 0.30647446732640765 - learning rate: 0.0001 - val loss: 0.4002171351629145 at epoch: 249: 100%| Loss: 0.30593346250789794 - learning rate: 0.0001 - val loss: 0.4002171351629145 at epoch: 250: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000250.pth Loss: 0.30605353545943065 - learning rate: 0.0001 - val loss: 0.3236594989019282 at epoch: 250: 100%| Loss: 0.31523266081364415 - learning rate: 0.0001 - val loss: 0.31214223188512463 at epoch: 251: 100% Loss: 0.3121627108239396 - learning rate: 0.0001 - val loss: 0.32304346386124105 at epoch: 252: 100%| Loss: 0.31201716769107385 - learning rate: 0.0001 - val loss: 0.40594975737964406 at epoch: 253: 100% Loss: 0.3132015399354272 - learning rate: 0.0001 - val loss: 0.29918953425744 at epoch: 254: 100%|█| Loss: 0.30866048826352177 - learning rate: 0.0001 - val loss: 0.29918953425744 at epoch: 255: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000255.pth Loss: 0.3088879721600308 - learning rate: 0.0001 - val loss: 0.3033159322598401 at epoch: 255: 100%|█ Loss: 0.30523368271491186 - learning rate: 0.0001 - val loss: 0.32445379215128284 at epoch: 256: 100% Loss: 0.3082592141304056 - learning rate: 0.0001 - val loss: 0.2987316995859146 at epoch: 257: 100%|█ Loss: 0.3077013487596392 - learning rate: 0.0001 - val loss: 0.32674457921701316 at epoch: 258: 100%| Loss: 0.31010557551098 - learning rate: 0.0001 - val loss: 0.3030623881255879 at epoch: 259: 100%|█| Loss: 0.3065339962239039 - learning rate: 0.0001 - val loss: 0.3030623881255879 at epoch: 260: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000260.pth Loss: 0.30648479753042696 - learning rate: 0.0001 - val loss: 0.2945685412953882 at epoch: 260: 100%| Loss: 0.3041540330465535 - learning rate: 0.0001 - val loss: 0.30476875252583446 at epoch: 261: 100%| Loss: 0.3068798369983915 - learning rate: 0.0001 - val loss: 0.31297055938664603 at epoch: 262: 100%| Loss: 0.30861558841029635 - learning rate: 0.0001 - val loss: 0.31001174186958985 at epoch: 263: 100% Loss: 0.31007959642752775 - learning rate: 0.0001 - val loss: 0.31483353060834546 at epoch: 264: 100% Loss: 0.307805921882391 - learning rate: 0.0001 - val loss: 0.31483353060834546 at epoch: 265: 100%|▉Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000265.pth Loss: 0.3080047332218003 - learning rate: 0.0001 - val loss: 0.3080143954823999 at epoch: 265: 100%|█ Loss: 0.30392203399992057 - learning rate: 0.0001 - val loss: 0.29799437347580404 at epoch: 266: 100% Loss: 0.3053085238281819 - learning rate: 0.0001 - val loss: 0.3291468602769515 at epoch: 267: 100%|█ Loss: 0.30717476932240995 - learning rate: 5e-05 - val loss: 0.30195652737337 at epoch: 268: 100%|█| Loss: 0.3009409159214094 - learning rate: 5e-05 - val loss: 0.2819372073692434 at epoch: 269: 100%|█| Loss: 0.3001459057770628 - learning rate: 5e-05 - val loss: 0.2819372073692434 at epoch: 270: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000270.pth Loss: 0.3000148224074139 - learning rate: 5e-05 - val loss: 0.30267828615272746 at epoch: 270: 100%|█ Loss: 0.2992089938793222 - learning rate: 5e-05 - val loss: 0.30576135218143463 at epoch: 271: 100%|█ Loss: 0.297558757301155 - learning rate: 5e-05 - val loss: 0.31899135428316455 at epoch: 272: 100%|█| Loss: 0.297383855617861 - learning rate: 5e-05 - val loss: 0.30306536835782666 at epoch: 273: 100%|█| Loss: 0.2956093915503062 - learning rate: 5e-05 - val loss: 0.29790930537616506 at epoch: 274: 100%|█ Loss: 0.30059484900209493 - learning rate: 5e-05 - val loss: 0.29790930537616506 at epoch: 275: 100%|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000275.pth Loss: 0.30052305703964527 - learning rate: 5e-05 - val loss: 0.3061572313308716 at epoch: 275: 100%|█ Loss: 0.3003741754647578 - learning rate: 5e-05 - val loss: 0.29603279601125154 at epoch: 276: 100%|█ Loss: 0.299627613510048 - learning rate: 5e-05 - val loss: 0.28398248027352724 at epoch: 277: 100%|█| Loss: 0.3021080169301984 - learning rate: 5e-05 - val loss: 0.3111071569078109 at epoch: 278: 100%|█| Loss: 0.2939733659206906 - learning rate: 5e-05 - val loss: 0.2877907016698052 at epoch: 279: 100%|█| Loss: 0.298268929477677 - learning rate: 5e-05 - val loss: 0.2877907016698052 at epoch: 280: 100%|▉| Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000280.pth Loss: 0.2983553654188225 - learning rate: 5e-05 - val loss: 0.30654799675240235 at epoch: 280: 100%|█ Loss: 0.2983045185626468 - learning rate: 5e-05 - val loss: 0.3073072275694679 at epoch: 281: 100%|█| Loss: 0.2977428478052427 - learning rate: 5e-05 - val loss: 0.29910268678384666 at epoch: 282: 100%|█ Loss: 0.2954770494746031 - learning rate: 5e-05 - val loss: 0.3032520132906297 at epoch: 283: 100%|█| Loss: 0.2962064707919311 - learning rate: 5e-05 - val loss: 0.3157319461598116 at epoch: 284: 100%|█| Loss: 0.2965031956535811 - learning rate: 5e-05 - val loss: 0.3157319461598116 at epoch: 285: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000285.pth Loss: 0.29662289303133843 - learning rate: 5e-05 - val loss: 0.29873434760991263 at epoch: 285: 100%| Loss: 0.29265239134410104 - learning rate: 5e-05 - val loss: 0.32044096378719106 at epoch: 286: 100%| Loss: 0.29690624289266565 - learning rate: 5e-05 - val loss: 0.29064199416076436 at epoch: 287: 100%| Loss: 0.292572217319301 - learning rate: 5e-05 - val loss: 0.2932455557234147 at epoch: 288: 100%|█| Loss: 0.2910314104214539 - learning rate: 5e-05 - val loss: 0.3159342232872458 at epoch: 289: 100%|█| Loss: 0.2917812002104754 - learning rate: 5e-05 - val loss: 0.3159342232872458 at epoch: 290: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000290.pth Loss: 0.2919222831809205 - learning rate: 5e-05 - val loss: 0.3200300826745875 at epoch: 290: 100%|█| Loss: 0.2920384829554764 - learning rate: 5e-05 - val loss: 0.2998523282654145 at epoch: 291: 100%|█| Loss: 0.2966673892911009 - learning rate: 5e-05 - val loss: 0.2861257344484329 at epoch: 292: 100%|█| Loss: 0.29242345933884256 - learning rate: 5e-05 - val loss: 0.3010793976923999 at epoch: 293: 100%|█ Loss: 0.29116458153791175 - learning rate: 5e-05 - val loss: 0.29494749184917 at epoch: 294: 100%|█| Loss: 0.3043915748845932 - learning rate: 5e-05 - val loss: 0.29494749184917 at epoch: 295: 100%|▉| 7Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000295.pth Loss: 0.3045244020730714 - learning rate: 5e-05 - val loss: 0.29799241002868204 at epoch: 295: 100%|█ Loss: 0.29013394646481655 - learning rate: 5e-05 - val loss: 0.2721177050295998 at epoch: 296: 100%|█ Loss: 0.2957085180648367 - learning rate: 5e-05 - val loss: 0.2805605329134885 at epoch: 297: 100%|█| Loss: 0.29426202751602587 - learning rate: 5e-05 - val loss: 0.30168647012289834 at epoch: 298: 100%| Loss: 0.2920073376789253 - learning rate: 5e-05 - val loss: 0.2961472237811369 at epoch: 299: 100%|█| Loss: 0.2952373994087707 - learning rate: 5e-05 - val loss: 0.2961472237811369 at epoch: 300: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000300.pth Loss: 0.29522111856039596 - learning rate: 5e-05 - val loss: 0.28166491494459267 at epoch: 300: 100%| Loss: 0.29718377228145654 - learning rate: 5e-05 - val loss: 0.32492024057051716 at epoch: 301: 100%| Loss: 0.29203436547575967 - learning rate: 5e-05 - val loss: 0.29104062125963326 at epoch: 302: 100%| Loss: 0.29241493915829864 - learning rate: 5e-05 - val loss: 0.2915509176604888 at epoch: 303: 100%|█ Loss: 0.2910756430117156 - learning rate: 5e-05 - val loss: 0.3040669306236155 at epoch: 304: 100%|█| Loss: 0.2957818931665833 - learning rate: 5e-05 - val loss: 0.3040669306236155 at epoch: 305: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000305.pth Loss: 0.2959351596348455 - learning rate: 5e-05 - val loss: 0.3111670963904437 at epoch: 305: 100%|█| Loss: 0.29206380135843446 - learning rate: 5e-05 - val loss: 0.3021305343684028 at epoch: 306: 100%|█ Loss: 0.2906969288361455 - learning rate: 5e-05 - val loss: 0.30570460943614736 at epoch: 307: 100%|█ Loss: 0.2900724932447471 - learning rate: 5e-05 - val loss: 0.3324916222516228 at epoch: 308: 100%|█| Loss: 0.29348984774269654 - learning rate: 5e-05 - val loss: 0.3073321966563954 at epoch: 309: 100%|█ Loss: 0.2912182815622684 - learning rate: 5e-05 - val loss: 0.3073321966563954 at epoch: 310: 100%|▉|Saved checkpoint: checkpoints/expert-discriminator-lr-1e-4-bs-64/checkpoint_192x192_epoch000000310.pth Loss: 0.29140155960908326 - learning rate: 5e-05 - val loss: 0.3095227970796473 at epoch: 310: 100%|█ Loss: 0.28802871273900676 - learning rate: 5e-05 - val loss: 0.3084297224002726 at epoch: 311: 100%|█ Loss: 0.2929448010698688 - learning rate: 5e-05 - val loss: 0.3082868711036794 at epoch: 312: 100%|█

I have some problems in training the syncnet expert !! Could you share your code, if you are possible? Thank you :)

[sahilg06]

Hi @Mayur28, I am facing the same problem. Can you please suggest the changes, that will lead to convergence? It will be a great help. Thanks!

[NikitaKononov]

@voducman Hello, how many videos there were in your training dataset? And what train / test splitting percentage did you use?

[liangleikun]

Hello, I have the same problem. I used LRS2 dataset to train lip_sync_net, but the loss does not go down. It is 0.68. Could you give me some advice?

[myme5261314]

Hello, I have the same problem. I used LRS2 dataset to train lip_sync_net, but the loss does not go down. It is 0.68. Could you give me some advice?

Using batch size 64 will dramatically make the convergence happen at early epochs. Cache the calculated melspectrogram into RAM will accelerate the training speed multiple times. You can use some code change snippets from my fork repo branch.

[purushottam0112]

Hii, Can any one please tell me, The LRS2 data set in which language and what exactly I have to write in filelists/ train.txt, val.txt file. I am trying to create my own dataset for finetune the model.

[purushottam0112]

Hi, @Mayur28 ,

I also fine tuned the lip-sync expert network in my own dataset, although it's has converged to lower than ~0.25, but when I use this pretrained discrimator to train the Waw2Lip network, I can not get the good results. Indeed, I could get the a visually good-looking face images, but the lip motion was completely not synced with the audio.

So I wonder whether there are some metrics to evaluate the lip-sync expert performance rather than just look at the loss.

Hii, Can you please tell me, The LRS2 data set in which language and what exactly I have to write in filelists/ train.txt, val.txt file. I am trying to create my own dataset for finetune the model.