DOI: 10.1162/089976698300017746

[refraction-ray]

• Link
• Title: Natural Gradient Works Efficiently in Learning
• Keywords (optional):
• Authors (optional):
• Reason (optional): NG is the same as SR in VMC, and can somehow be approximated by Adam
• Summary (optional):
• Ideas (optional):

[refraction-ray]

Relevant works on natural gradient

• Revisiting natural gradient for deep networks relation with Hessian free methods
• New insights and perspectives on the natural gradient method relation between Fisher matrix and Hessian
• Accelerating Natural Gradient with Higher-Order Invariance
• Optimization Methods for Large-Scale Machine Learning A great review on optimizers that especially suitable for DL set up, see 6.4 for natural gradient analysis
• Limitations of the Empirical Fisher Approximation empirical Fish may be not connected to real Fisher
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature KFAC, an efficient way to approximate natural gradient
• Pathological spectra of the Fisher information metric and its variants in deep neural networks singularity on Fisher matrix, also applied to SR method
• Noisy Natural Gradient as Variational Inference relation between NG and its role in VI

[refraction-ray]

Other resources on natural gradient

• Post on Fisher information and post on natural gradient
• Post on NG and its implementation in TensorFlow
• Post on KFAC

[refraction-ray]

Comment: I am not independently realizing that stochastic reconfiguration method in VMC is the same thing as natural gradient descent. The idea is from this paper: PHYSICAL REVIEW X 8, 011006 (2018). This is a physics paper on RBM wavefunction VMC, however, the authors mentioned in this work:

stochastic reconfiguration optimization, which is equivalent to the natural gradient descent

Great observation! And it is impressive that some optimization based on imaginary time dynamics and Schrodinger equations can be fully reduced to general optimization setups. But I am not sure who is the first one that realized this connection.

Idea: If I have time, I will think about imaginary time evolution in TNS settings, which is also used for optimization ground state energy. And I will definitely check whether such "physicsy" optimization scheme can also be reduced to some general well-established optimization scheme in optimization field.

Update: this relation is in particular be clarified in Append B of arXiv:1909.02487. But the authors mentioned in this work that either community doesn't realize it before. No, the above PRX is the evidence. Another paper on this: On the geometry of learning neural quantum states