search

TyXe-BDL / TyXe

MIT License
144 stars 33 forks source link

Important data leakage in resnet example. #21

Closed Cam-B04 closed 1 year ago

Cam-B04 commented 2 years ago

Hi !

First thank you very much for this repo, it is very helpful for people who are new to BNNs like me.

I have started to use TyXe for a convolutional BNN, starting from your resnet.py example. After getting some unexpected behavior during training I have noticed that in the callback function the network was not set into evalutation mode. It is resulting in an important data leakage by training the network on the test dataset as well if I am correct.

I would recommend to start and finish the callback function with respectively b.eval() and b.train().

Hope this helps, Regards!

hpplyt commented 1 year ago

Thanks spotting this and the PR! I don't think it's overkill at all, I'll merge it right away, sorry for only getting back to you now.

Technically I wouldn't really call this data leakage since training won't be affected by the test data, but it does break independence between the test predictions, so I definitely should have included the eval and train calls :)