# LDS should be calculated before the forward for cross entropy

lyakaap / VAT-pytorch

Virtual Adversarial Training (VAT) implementation for PyTorch

296 stars 43 forks source link

# LDS should be calculated before the forward for cross entropy #6

Closed jizongFox closed 4 years ago

jizongFox commented 5 years ago

Hi, I am wondering why you commented " # LDS should be calculated before the forward for cross entropy". What would be the specific reason for that?

jonkoi commented 5 years ago

The only thing I can think of is that the the repo owner wants to get lds before running stats of the batch norms get changed by forwarding.

zdhNarsil commented 4 years ago

But in VATLoss the author use _disable_tracking_bn_stats so the BN statistics shouldn't be a problem. I am still curious why " # LDS should be calculated before the forward for cross entropy".