search

vislearn / FrEIA

Framework for Easily Invertible Architectures
MIT License
775 stars 113 forks source link

Negative Errors When Training #16

Closed ayaz-amin closed 4 years ago

ayaz-amin commented 4 years ago

Hello there! I am training an INN on the CIFAR10 dataset, and everything works smoothly. However, I noticed that when I use the NLL from the examples, I get negative errors that continually decrease with each epoch. Here is the loss function that I used:

...
loss = torch.mean(z**2) / 2 - torch.mean(log_det) / 32*32
loss.backward()
...

Although training works fine, I get errors in the negative range. Even when I modified it to give me positive errors, it will converge to the negative spectrum. Is there something wrong with the code?

ardizzone commented 4 years ago

Hi!

This question comes up often for normalizing flows and similar models using NLL! Indeed, the loss is supposed to be negative, it is not a problem with your code. (There is however a mistake I spotted, it should be / (32*32*3) as far as I can see). To double check: for CIFAR10 the NLL should be somewhere between -1 and -5.

Let me know if that answers the question!

ayaz-amin commented 4 years ago

Thanks!

ayaz-amin commented 4 years ago

So, I am getting losses around the -30000 range. Is this considered stable?