search

tensorflow / probability

Probabilistic reasoning and statistical analysis in TensorFlow
https://www.tensorflow.org/probability/
Apache License 2.0
4.16k stars 1.08k forks source link

Normal Inverse Gaussian Outputs Positive log_prob #1782

Closed i418c closed 5 months ago

i418c commented 5 months ago

While training a model with a NIG output, I noticed that the loss for the negative log likelihood would go negative. At first I thought that perhaps the values of verify in the model were outside of support for the NIG function, but fixing these values to what (I think) should be valid still would yield a negative loss.

Any help on why this occurs and how to fix it would be appreciated.

While creating an example, I noticed that the quickest way to trigger this was to set the verify values to single number. A gist reproducing the issue can be found here.

csuter commented 5 months ago

This is fine. Probability densities can be large and positive, yielding positive log prob and negative NLL loss.

On Sun, Jan 14, 2024 at 23:44 i418c @.***> wrote:

While training a model with a NIG output, I noticed that the loss for the negative log likelihood would go negative. At first I thought that perhaps the values of verify in the model were outside of support for the NIG function, but fixing these values to what (I think) should be valid still would yield a negative loss.

Any help on why this occurs and how to fix it would be appreciated.

While creating an example, I noticed that the quickest way to trigger this was to set the verify values to single number. A gist reproducing the issue can be found here https://gist.github.com/i418c/8e407534558ede92e39e2fbfc5e9c91f#file-normal-inverse-gaussian-negative-log_prob-ipynb .

— Reply to this email directly, view it on GitHub https://github.com/tensorflow/probability/issues/1782, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABG2GM3JJBB2E5B63AR5NDYOSX37AVCNFSM6AAAAABB2VQMFOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA4DCMRUGA4DANQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

i418c commented 5 months ago

Let me ask a different question then. If the loss of my model can be negative, how would I know if it's anywhere near optimal? Experimentation?

csuter commented 5 months ago

Hehe, welcome to the wild world of high dimensional nonconvex optimization 🙂. No idea! I think most people abandon hope of provable optimality in most cases and focus on whether a particular method solves a specific problem satisfactorily.

On Mon, Jan 15, 2024 at 09:28 i418c @.***> wrote:

Let me ask a different question then. If the loss of my model can be negative, how would I know if it's anywhere near optimal? Experimentation?

— Reply to this email directly, view it on GitHub https://github.com/tensorflow/probability/issues/1782#issuecomment-1892278341, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABG2GJJ7MKJJBPARNDK36TYOU4J3AVCNFSM6AAAAABB2VQMFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJSGI3TQMZUGE . You are receiving this because you commented.Message ID: @.***>

i418c commented 5 months ago

Thanks for the help. It may be helpful for the project to have a tutorial or example to highlight that a negative loss isn't abnormal when training with NLL. Those of us who did poorly in statistics would be grateful.