search

timgaripov / swa

Stochastic Weight Averaging in PyTorch
BSD 2-Clause "Simplified" License
966 stars 129 forks source link

SWA with Torchbearer #23

Open AnabetsyR opened 2 years ago

AnabetsyR commented 2 years ago

Hi! I am trying to implement SWA with Torchbearer but I'm having issues with the state etc. Could anybody point me in the direction of some resources to better understand how to use SWA with Torchbearer, particularly the Trial class?

Thank you in advance! Anabetsy

izmailovpavel commented 2 years ago

Hi @AnabetsyR, sorry for the late reply!

I have never used Torchbearer, from their github page it seems like it is no longer maintained, and the successor is pytorch lightning. Lightning actually has its own implementation of SWA.

Generally, you may also want to try using the native pytorch implementation of SWA, but I don't know how easy it is to use it with Torchbearer.

AnabetsyR commented 2 years ago

Hi @izmailovpavel! Thank you so much for taking the time to get back to me! I considered both the native Pytorch implementation and the Lightning one. But because the rest of my code uses some usefull callbacks in Torchbearer, I was trying to avoid a full change. I wrote the Torchbearer guys and one of them taught me how to pass the native Pytorch swa as a callback to my Torchbearer implementation. I'm testing it now but I'm hopeful.

P.S. I loved reading your paper on SWA.

Thank you so much again!

izmailovpavel commented 2 years ago

Happy to hear that you liked the paper :)

Sounds great, I hope that solution works out!