Paper reading

[mekty2012]

Write simple summary of papers you read, and how can it helps our project.

What you must read in paper

1. What is target problem, what is model they used.
2. Background and Related Work

[mekty2012]

Paper name : Inverse problems in approximate uniformm generation is

Target : Given boolean function f:B^n->B, generative model that returns some b, s.t. f(b)=1

Method : ???

Help to project : None, however will be helpful if we change to boolean function learning.

[mekty2012]

Paper name : Conditional Sampling from Invertible Generative Models with Applications to Inverse Problems

Target : Given measurements y, measurement matrix A, prior distribution p(x), conditional distribution p(x|Ax=y).

Method : Real valued NVP https://arxiv.org/pdf/1605.08803.pdf

Help to project : target problem might be helpful. However, used dataset is MNIST and CelebA-HQ, having 2828, 6464 dimensions.

[mekty2012]

Paper name : Solving Inverse Problems with Piecewise Linear Estimators : from Gaussian Mixture Models to Structured Sparsity

Target : Image inversion, meaning given some linear non-invertible degradation operator U, noise w, y=Uf+w where f is image, inverting f from y.

Method : Create local patches of images, view them as GMM. Compute parameters by MAP-EM algorithm. Requires estimating parameters from the degraded data.

Help to project : We can use high dimensional data by adopting idea of local patch!

TODO : learn more about sparse inversion.

[Lee-Janggun]

Paper name: Solving Inverse Porblmes by Joint Posterior Maximization with a VAE Prior .

Target: Image inversion.

Method: Simplify calculations for Explicitly decoupled priors by calculating the posterior p(x,z|y), instead of p(x|y), z latent variable, and have statistical encoders and decoders. Showed for VAE, which had good convexity for ensuring convergence. Used this to en/decode MNIST handwritten numbers with VAE.

Help to project: If VAE were to be used in project, may be referenced for better speed. However need to know VAE first.

TODO: learn VAE

[mekty2012]

TODO: learn VAE

VAE(Variational Auto Encoder) is neural network model that performs dimensionality reduction and generative modeling. It do not require target label, so you can view it as unsupervised learning.

The main idea is viewing some data, like image data is over some distribution over latent variable with lower dimension.

It consists 2 networks, namely encoder network and decoder network.

First, encoder network is given input data, returns two output, mean of z and standard deviation of z.

Then we sample from standard normal distribution, multiply it to standard deviation computed right before. This creates noise on latent variable, so that model runs well on harder condition.

Then with noised latent variable, decoder inverts what encoder does, sampling original data from latent variable. The total network is trained over distance between original data and sampled data.

For more, I recommend https://ratsgo.github.io/generative%20model/2018/01/27/VAE/.

Edit : The basis of VAE is autoencoder. Maybe learning autoencoder may help you understanding this concept. The main difference between VAE and autoencoder is that VAE uses variational inference. Also, if you remember, the purpose of VAE is same to GAN(Generative Adversarial Modeling).

[Lee-Janggun]

Paper name: Variational Inference for Computational Imaging Inverse Problems

Target: Inverse Problems for computational imaging

Method: In training the forward function, under a proposed approximation p(y'|x) train p(y|x) ,using a multi-fidelity model, with much less samples smartly. However the detour does seem to come at a complexity cost due to the creation of many new distributions.

Help to Project: If can be successfully used, increase speed of our method, but does seem quite complicated.

Side note: The paper seemed to be very excessive in explaining basic details of ML in the beginning, and also referenced a lot of other papers. The appendix included more details of the architecture that the proposed model was applied.

[mekty2012]

Paper name : High dimensional data clustering

Target : Object localization

Other methods : The classical application of data clustering on high dimensional data is applying dimensionality reduction methods, like PCA, kernel PCA, non-linear PCA. Finding subspaces are usually done by some heuristics, by grid searching and dense clusterable subspaces.

Method : This paper describes applying GMM on high dimensional data (We usually say data is high dimensional if n << d where n is number of data and d is number of parameter). The problem here is that covariance matrix is too large, since it requires n*n values. To overcome this, this model assumes that most of eigenvalues of covariance matrix is small, and equal with exception of few. The method presented here uses EM algorithm.

Help to Projects : This paper gives how to apply high dimensional data on GMM like image data.