Implement Laplace (quadratic) approximation

carsten-j commented 1 month ago

This is an early version of q quadratic approximation implementation that I have developed while reading Statistical Rethinking by Richard McElreath.

There is a short discussion about this in the issue and maybe @theorashid can help with feedback of this draft PR.

This work is partly based on the Python package pymc3-quap but pymc3-quap is based on PYMC3 and a lot happend bewteen version 3 and 5 of PYMC. Optimizers works better when provided with a good initial guess and hence a (optional) starting point has been added to function arguments. Please see Github for a discussion about the differences between PYMC version 3 and 5 for computing the Hessian.

carsten-j commented 1 month ago

I am looking for the best way to return not just a posterior sample distribution but also the mean vector and covariance matrix of the Gaussian distribution. Any suggestion for this. So far my only idea is to add another section to the inferenceData returned containing this information. Thoughts on this?

zaxtax commented 1 month ago

I'm not sure the InferenceData is the best place to put it. We should copy whatever we do with Variational Inference

On Sun, 2 Jun 2024, 13:05 Carsten Jørgensen, @.***> wrote:

I am looking for the best way to return not just a posterior sample distribution but also the mean vector and covariance matrix of the Gaussian distribution. Any suggestion for this. So far my only idea is to add another section to the inferenceData returned containing this information. Thoughts on this?

— Reply to this email directly, view it on GitHub https://github.com/pymc-devs/pymc-experimental/pull/345#issuecomment-2143803800, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAACCUM45EIPXABROOJPVCTZFL353AVCNFSM6AAAAABIT3F2F2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTHAYDGOBQGA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

aloctavodia commented 1 month ago

Historically, Inferencedata has been focused on mcmc. But we have discussed a few times extend it to better handle other inference methods, like SMC or variational methods. It just that there has not been enough momentum to agree and implement and schema that works for those methods.

carsten-j commented 1 month ago

@zaxtax and @aloctavodia are you saying that I should not return inferencedata at all or just not return the gaussian mean and covariance in the inferencedata object? I am new to both PYMC and Bayesian statistics so I do not know the history of this package. Best, Carsten

zaxtax commented 1 month ago

Oh, it's more that we haven't decided how to handle this within the library. Don't treat this as a blocker, though we should raise it for discussion more broadly

On Sun, 2 Jun 2024, 19:04 Carsten Jørgensen, @.***> wrote:

@zaxtax https://github.com/zaxtax and @aloctavodia https://github.com/aloctavodia are you saying that I should not return inferencedata at all or just not return the gaussian mean and covariance in the inferencedata object? I am new to both PYMC and Bayesian statistics so I do not know the history of this package. Best, Carsten

— Reply to this email directly, view it on GitHub https://github.com/pymc-devs/pymc-experimental/pull/345#issuecomment-2143944495, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAACCUKGG7KIW4SVLMG6ZLLZFNGB7AVCNFSM6AAAAABIT3F2F2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTHE2DINBZGU . You are receiving this because you were mentioned.Message ID: @.***>

twiecki commented 1 month ago

CC @ferrine

aloctavodia commented 1 month ago

Oh, it's more that we haven't decided how to handle this within the library. Don't treat this as a blocker, though we should raise it for discussion more broadly

exactly, just saying that if necessary InferenceData can be extended.

ricardoV94 commented 1 month ago

Suggestion, include two groups in the returned inferencedata:

A fit group that includes the mean and covariance of the laplace fit, and
a posterior group that includes draws from this fit, and has all the bells and whistles like dimensions, deterministics, etc... Include the default extra groups like observed and constant data. This will look just like a fit from mcmc sampling. This can be disabled by the user by setting draws = 0

We could even try different fits from distinct initialization points (optionally) and save those as distinct "chains" in the fit and corresponding posterior groups. Although usually multiple initialization are used with the goal of finding the best fit, they could still be useful to detect multi-modality / pervasiveness of local optima.

ricardoV94 commented 1 month ago

@carsten-j PR looks great! I left some comment above

carsten-j commented 4 weeks ago

Thanks you @ricardoV94 and @twiecki for the review comments. I believe that all of them expect one has been fixed. I have not figured out how to use remove_value_transforms. I tried to browse through PYMC source code but that did not really help.

ricardoV94 commented 3 weeks ago

Thanks you @ricardoV94 and @twiecki for the review comments. I believe that all of them expect one has been fixed. I have not figured out how to use remove_value_transforms. I tried to browse through PYMC source code but that did not really help.

The docs contains code example: https://www.pymc.io/projects/docs/en/stable/api/model/generated/pymc.model.transform.conditioning.remove_value_transforms.html