Neural Process

[JisuHann]

Neural Process

1. Conditional Neural Processes(ICML2018) paper
2. Neural Processes(ICML2018) paper
3. Convolutional Conditional Neural Processes(ICLR2020) paper

CNP: Conditional Neural Processes(ICML2018)

• define conditional distributions over functions given a set of observations

  1. a family of neural models (that represents solutions to the supervised problem) -> embedding h
     • trained by sampling a random dataset
  2. end-to-end training approach to learning them, that aggregate NN (using GP) -> aggregator r
  3. conditions the function g

• Advantage

  • make accurate predictions after observing only a handful of training data points -> scale to complex functions and large datasets

• On classification task

  • parametrizes the logits of the class probabilities p_c over the c classes of a categorical distributions
  • Apply on Omniglot dataset for one0shot classification

LNP: (Latent) Neural Processes(ICML2018)

• a class of neural latent variable models

GP, DNN, Meta-learning

• Gaussian Process
  • design appropriate priors
  • on CNP, use flexibility of stochastic processes
    • ability to extract prior knowledge directly from training data tied with scalability at test time
    • models that directly parameterize conditional stochastic processes
  • on CNP, not implement Bayesian inference directly and not true that the conditional distributions will be consistent with respect to some prior process
• Deep Neural Networks
  • on CNP, structured as DNN

[JisuHann]

ConvCNP: Convolutional Conditional Neural Process (ICLR2020)

• NP family that models Translation Equivariance functions/functional representations on sets, it can naturally be extended to handle varying-size sets
  • Model embeds data sets into an infinite-dimensional function space as opposed to a finite-dimensional vector
• off-the-grid(e.g. image pixels form a regularly spaced grid, (GP datasets))
• on-the-grid(e.g. time series data may be observed irregularly at any time,(MNIST, CelebA32))

  ConvCNP: CNP + (Permutation-invariant, Translation Equivariance)

• Permutation-invariant: output doesn't change due to the permutation --> 이미 정의됨
  • The form employed by the NP family for the encoder that embeds sets into a latent representation.
• Focus - Translation Equivariance: If the input locations of the data are translated by an amount, the predictions should be translated correspondingly.

1. Convolutional Deep sets

• Translation Equivariance with respect to translations on X
• Enrich the encoder E:Z->H to map into a function space H containing functions on X

  Definition 1. Functional mappings on sets and functional representations of sets:: Encoder of ConvCNP

  a continuous function satisfies Properties 1,2(Permutation-invariant, Translation Equivariance) if and only if it has a representation of the form

  

Definition 2. Multiplicity

ex. we often observe only one (possibly multi-dimensional) observation per input location, which corresponds to multiplicity one.

Theorem 1

a function is continuous, permutation invariant and translation equivariant if and only if it has a representation of the form

for some continuous and translation-equivariant \rho and some continuous \phi and \psi , where H is an appropriate space of functions that include the image of E. We call a function \Phii of the above form CONVDEEPSET.

2. Convolutional Conditional Neural Processes

Form of \phi

• Density channel, normalized convolution

  ConvCNPs for off-the-grid data

ConvCNPs for on-the-grid data

[JisuHann]

ConvLNP: Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes

[JisuHann]

SNP: Sequential Neural Processes

[JisuHann]

ASNP: Attentive Sequential Neural Processes

[JisuHann]

RMR: Robustifying Sequential Neural Processes