search

jankrepl / mildlyoverfitted

Paper implementations from scratch and machine learning tutorials
MIT License
340 stars 124 forks source link

Unable to verify Vision Transformer result using loaded weight from timm #23

Closed wanburana closed 1 year ago

wanburana commented 1 year ago

I tried to run this file https://github.com/jankrepl/mildlyoverfitted/blob/master/github_adventures/vision_transformer/verify.py to reproduce the result, the last assertion is not passed. The result from the official model and the custom model is not the same, eventhough the weights are exactly the same, im not sure why. Also, it used to work flawlessly, I suspect it is something to do with torch 2.0 (or maybe timm), but im not sure.

here is the quick runnable notebook to verify the issue https://colab.research.google.com/drive/1sRyBTvqAMkTgmSWlBFpv79mj6Ir4aLOT?usp=sharing

jankrepl commented 1 year ago

@wanburana I had a quick look at your rendered notebook

....
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 874 / 1000 (87.4%)
Max absolute difference: 1.9073486e-06
Max relative difference: 0.00248577

The differences are really small or I would even say negligible.

But yeh, something must have changed since the video was released. Not sure if I will have time to look deep into it

wanburana commented 1 year ago

Thank you, yeah. After looking closely it's not a big deal at all. Still, it was weird how two exactly same weights multiplied producing different result.

jankrepl commented 1 year ago

:)) Hehe, who knows:)

I am closing the issue. In case you change your mind and want me (or someone else) to dig deeper into this feel free to reopen:)

Thank you for you interest @wanburana