greta-dev / greta

simple and scalable statistical modelling in R
https://greta-stats.org
Other
524 stars 63 forks source link

Update optimis(z)er interface to TF2 #551

Closed njtierney closed 2 weeks ago

njtierney commented 1 year ago

As described by @goldingn in #306 - here are some of the changes

This issue is about planning ahead to dealing with major TensorFlow API changes when a greta version has to work with TensorFlow 2.0. It doesn't affect anything yet.

optimisers

Tensorflow's various optimiser interfaces are being unified (details here). Some of the old versions will still be available in the compat compatibility module (though there may be speed benefit to changing to more recent versions). In addition, the TF 1.x contrib module will no longer be available in TF 2.0, and won't be in compat.

There is a tensorflow addons repo with an 'optimizers' module (though with no apparent overlap with existing optimisers) and a TFP 'optimizer' module which has a couple of the methods.

greta optimiser TF 1.x function TF2.0 function
nelder_mead tf$contrib$opt$ScipyOptimizerInterface tfp$optimizer$nelder_mead_minimize
powell tf$contrib$opt$ScipyOptimizerInterface ?
cg tf$contrib$opt$ScipyOptimizerInterface ?
bfgs tf$contrib$opt$ScipyOptimizerInterface tfp$optimizer$bfgs_minimize
newton_cg tf$contrib$opt$ScipyOptimizerInterface ?
l_bfgs_b tf$contrib$opt$ScipyOptimizerInterface ?
tnc tf$contrib$opt$ScipyOptimizerInterface ?
cobyla tf$contrib$opt$ScipyOptimizerInterface ?
slsqp tf$contrib$opt$ScipyOptimizerInterface ?
gradient_descent tf$compat$v1$train$GradientDescentOptimizer tf$keras$optimizers$SGD
adadelta tf$compat$v1$train$AdadeltaOptimizer tf$keras$optimizers$Adadelta
adagrad tf$compat$v1$train$AdagradOptimizer tf$keras$optimizers$Adagrad
adagrad_da tf$compat$v1$train$AdagradDAOptimizer tf$compat$v1$train$AdagradDAOptimizer
momentum tf$compat$v1$train$MomentumOptimizer tf$keras$optimizers$SGD
adam tf$compat$v1$train$AdamOptimizer tf$keras$optimizers$Adam
ftrl tf$compat$v1$train$FtrlOptimizer tf$keras$optimizers$FTRL
proximal_gradient_descent tf$compat$v1$train$ProximalGradientDescentOptimizer tf$compat$v1$train$ProximalGradientDescentOptimizer
proximal_adagrad tf$compat$v1$train$ProximalAdagradOptimizer tf$compat$v1$train$ProximalAdagradOptimizer
rms_prop tf$compat$v1$train$RMSPropOptimizer tf$keras$optimizers$RMSProp (though arguments may have changed, so tf$compat$v1$train$RMSPropOptimizer as a backup)

Unless another interface to the Scipy optimisers is developed, the optimisers without replacements will have to be removed from the API. It'll bee a little hard to deprecate them without knowing whether a replacement will be coming.

ode solvers

The greta.dynamics package's ode_solve() functionality also depends on contrib, using the tf$contrib$integrate$odeint and odeint_fixed methods. tfp$math$ode$Solver$solve may be a viable replacement.

njtierney commented 1 year ago

This has been resolved as of https://github.com/njtierney/greta/commit/eef698f23effbc97595d61c3377cdfc1c8686de1