Open marco-rudolph opened 2 years ago
@marco-rudolph well, flows can minimize KL-divergence up to the constant or, alternatively, can have log-likelihoods larger than 0.0 which is not desirable. So, to prevent this and to avoid negative loss I added sigmoid. In practice, it can help to improve performance a bit.
Why are likelihoods larger than 0.0 not desireable? Negative loss itself should not be a problem as long as it does not lead to instabilities which is not really the case here. But it is an interesting finding that this improves the performance :)
Hi Denis! According to your code, you use some "negative log sigmoid log likelihood loss": https://github.com/gudovskiy/cflow-ad/blob/4d6ec4719e836422c5253a68c1150ac172d8000d/train.py#L76 https://github.com/gudovskiy/cflow-ad/blob/4d6ec4719e836422c5253a68c1150ac172d8000d/train.py#L78 with https://github.com/gudovskiy/cflow-ad/blob/4d6ec4719e836422c5253a68c1150ac172d8000d/train.py#L18
What is the motivation behind using this kind of loss?
Thanks in advance, Marco