VBPCA

[pauldwkirk]

To do: add Ilin and Raiko version of VBPCA (equations 49 and 50 in their paper), by copying and modifying mpcaNetR.cpp.

Harry - if there are things I can do to help with this (e.g. annotating the mpcaNetR.cpp code more fully or writing a Matlab prototype), just let me know.

[HGray384]

I've implemented the parameter updates. Am I right in thinking that in order to update the log-likelihood, I will need to include the formula for q(W, X, m) in Appendix C page 1994 from Ilin and Raiko?

[pauldwkirk]

What we really need is the variational lower bound. I'm looking into it - will comment again soon.

[pauldwkirk]

(By the way: good point. I hadn't thought about this.)

[pauldwkirk]

I think we might need to derive this for ourselves. In the first instance, I think it would be easiest to do this in the case of no missing values, and then add in the extra complexity once we're comfortable with what we've got.

This might or might not be helpful, but there is some Matlab code accompanying the Ilin paper, here: https://users.ics.aalto.fi/alexilin/software/ In principle, the "cf_full.m" file should include a calculation of the variational lower bound (if we figure out which bits of the code are executed in the VB case). In practice, this might be more useful as a check, once we have derived the lower bound ourselves.

How would you feel about this? Have you done anything like this before (working through VB calculations)? No problem if not - I can help.

[HGray384]

I was hoping this wouldn't be the case!

No problem though, I had already downloaded their code since I noticed they have some of the initialisations for the extra hyperparameters in the VB case (no justifications however). I'm now familiar with their code so I'll have a dig into cf_full.m and translate what I can.

In the meantime can you point me in the direction of some good online resources for the VB maths please? I'll also push my current changes to ppcaNet to github so you can check out the parameter updates and tell me if I have gone wrong (I have only checked that the package still installs after my added functions, not that they actually do what we want yet).

[pauldwkirk]

Agreed - it's a pain.

For PCA specifically, take a look at: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bishop-vpca-icann-99.pdf

For variational Bayes more generally, Ch. 10 of the Bishop book (PRML) is good. Do you have my copy of that, btw? I don't need it - I just can't remember who I loaned it to.

[pauldwkirk]

I was thinking about this while cycling. For now, it might be quickest just to replicate the functionality of the Matlab code. That would be enough for the application note. Your call though -- just let me know if I can help with anything. Regarding the derivation, maybe we should both have a go, and then compare notes. Not sure whether or not I'll have chance to have a bash this week though.

[HGray384]

Yep this is a good plan. Will get back to you with a) matlab translation b) VB derivation

[HGray384]

EDIT: there are still bugs in the updates, but it does now run

Translation done, check the new function out.

[pauldwkirk]

Good work. I will have a play and compare calculated quantities to the Matlab code.

[HGray384]

EDIT: parameter initialisations are different, but results should still be able to be compared

Compare using the latest commit, it works now.