search

Photrek / Nonlinear-Statistical-Coupling

Apache License 2.0
5 stars 1 forks source link

Develop Cross-Entropy, Entropy, and Divergence #1

Closed kenricnelson closed 3 years ago

kenricnelson commented 3 years ago

Complete the development of Coupled Cross-Entropy, Coupled Entropy, and Coupled Divergence in Python Use the Mathematica functions in Coupled_Functions.nb as guides for the development Daniel Svoboda has prototyped the code, see his workspace folder and the file functions.py

The important points are: 1) Leverage the enhanced CoupledLogarithm function 2) Insure that both the original definition and the definition with the alpha root is implemented and verified against the Mathematica plots 3) Make use of the Hamiltonian Monte-Carlo integration method or a similar tool from the TFP MCMC folder 4) The TFP VI (Variational Inference) folder has divergence functions which can provide some guidance, but our interests are in having the three functions: cross-entropy, entropy, and divergence 5) Develop the Coupled Cross-Entropy first which has two distributions as inputs 6) Coupled Entropy should call Coupled Cross-Entropy with the same distribution 7) Coupled_Divergence = Coupled_Cross_Entropy - Coupled_Entropy; thus the divergence function will be implemented very differently than tfp vi library; however, this seems like the simplest way to ensure that there is one foundational function the cross-entropy which the others are dependent on.

kenricnelson commented 3 years ago

John utilized the quad integration to develop the entropy functions

kenricnelson commented 3 years ago

John utilized the quad integration to develop the entropy functions