Write a proper (K)PCA notebook

[karlnapf]

Illustrating what PCA does, with proper eye candy.

Some ideas:

• Explain the math of the method (both versions, Eigendecomposition or SVD), some background
• Gaussian pancakes of 3D Gaussian where PC is just noise. Plot the subspace and the projection into it.
• Plot the principal components and show how choosing different numbers of them affects the projections.
• PCA on real data (for example Eigenfaces?)
• All this for Kernel PCA illustrating the main differences (non-linear manifold rather than a linear subspace). What does the kernel choice do to the projections?
  • Have a toy example where N>>D and D>>N

This can be combined with many different dimension reduction methods such as Factor analysis, PPCA, etc. Creating cool looking examples is always fun!

Many examples and explanations (including the pancake idea) can be found in http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomePage

Or here http://www.gatsby.ucl.ac.uk/teaching/courses/ml1-2013/lect2-handout.pdf

[kislayabhi]

I think I'll take this. If there's any more idea or nuance you want to add related to above, please tell. :)

[karlnapf]

There is so much literature on this topic. In particular PCA, so this notebook should be nice and clear and full of cool plots :)

[karlnapf]

@kislayabhi Looks like your pca repository is well suited to be used as a ressource :)

[karlnapf]

See also #1915 and all its additions to the PCA class. Those should be documented with examples where the pros and cons of the methods are discussed

[kislayabhi]

ahh! I am stuck at using the pca for eigenfaces. The method preprocessor.init(datamatrix) couldn't handle even 100x100 pixels images.

[kislayabhi]

it stucks for quite a while and then system hangs.

[kislayabhi]

but when i resize the image to 50 x 50 pixels, it gives me the output after a while

[mazumdarparijat]

@kislayabhi ok! its taking huge memory I presume.

[kislayabhi]

yup! thats for sure!!

[mazumdarparijat]

@kislayabhi We recently added SVD based PCA as well. can you check with that once plz?

[kislayabhi]

@mazumdarparijat Is this added in the issue #1915 . I am afraid i didn't update my local repo!.

[mazumdarparijat]

yes! Will it be possible for you to share the link to your data? I am eager to try myself to see whats happening.

[kislayabhi]

I am using the face image data that comes along : path shogun/data/faces/oksanafaces*.pgm

[mazumdarparijat]

@kislayabhi please update your local repo and then try, it might work because we did a major overhauling of the class in #1915. If it still doesn't work, then its time to get our hands dirty again!! :)

[karlnapf]

The old PCA code decomposed the covariance matrix of the data. For 100x100 images, this is a 10000-dimensional square matrix, which causes problems when decomposing. @mazumdarparijat 's new SVD based PCA should solve this. As he said, computing the SVD of the data matrix should be cubic in the number of points, so that should work.

If it doesn't please post some simple python code (possibly on toy data) that illustrates the problem in an isolated way.

Thanks!

[karlnapf]

I just tried the new PCA implementation on datasets with

• D=100000, N=100
• N=100000, D=100

and both work quite fast plain vanilla (using the std constructor p=PCA()). So that works - and is a major improvement to what we had before :)

[kislayabhi]

@mazumdarparijat. yeah. Thanks Your update has made my life easy. Things are working as robust as they should. :) :) @karlnapf : yes, everything is working fast. thanks for making it clearer. :)

[mazumdarparijat]

@karlnapf @kislayabhi thats great news then! :)

[kislayabhi]

@karlnapf I have sent a PR! Just one thing! I have applied eigenfaces on att_dataset which the user may have to download from their side!!!

[vigsterkr]

@karlnapf are we done with this?

[karlnapf]

Kernel PCA is still missing, but just a small addition to the existing PCA one. @kislayabhi interested on adding some things? we have a graphical example that we could put in

[kislayabhi]

@karlnapf yeah sure man. Details about the example to be added?

[karlnapf]

Yeah sure, basically we want this here: http://scikit-learn.org/stable/auto_examples/decomposition/plot_kernel_pca.html

It might also be worth re-organising the KPCA code and make it have the same nice properties as your kick-ass PCA implementation, and also phrased again the linalg framework!

[karlnapf]

We long have a PCA notebook