search

tensorflow / probability

Probabilistic reasoning and statistical analysis in TensorFlow
https://www.tensorflow.org/probability/
Apache License 2.0
4.26k stars 1.1k forks source link

tfp.optimizer.lbfgs_minimize as Keras optimizer? #565

Open Hoeze opened 5 years ago

Hoeze commented 5 years ago

Hi, is there a way to use tfp.optimizer.lbfgs_minimize as Keras optimizer? This would be quite useful in certain cases where the loss function is approximately quadratic.

A colleague of mine would very much need it since an autoencoder written in R with negative-binomial loss converges faster than its Keras counterpart.

dynamicwebpaige commented 4 years ago

Thanks for the recommendation, @Hoeze! Are you using stand-alone Keras for your work, or tf.keras?

Hoeze commented 4 years ago

@dynamicwebpaige Thanks for your answer. We are usually using the default Keras API with Tensorflow as backend. For special needs, we construct Keras layers with the Tensorflow API.

I would be cool if it was implemented as a Tensorflow optimizer similar to ADAM: https://github.com/tensorflow/tensorflow/blob/r2.1/tensorflow/python/training/adam.py This way it would be more easy to use in plain Tensorflow as well, since switching to L-BFGS would only incorporate changing a single line of code.

The big problem with tfp.optimizer.lbfgs_minimize was that we had to provide it with a function returning the loss. Then it does some magic and eventually the result has converged. One cannot record the single optimization steps. I am not sure how this black-box optimization fits with Keras, since e.g. the EarlyStopping callback would not work this way.

dario-passos commented 4 years ago

@Hoeze I was looking for the same functionality and I found this blog that show how to use the lbfgs_minimize() with a tf.keras model: https://pychao.com/2019/11/02/optimize-tensorflow-keras-models-with-l-bfgs-from-tensorflow-probability/

However it would be very useful if the tf.keras development team could implement this as a Tensorflow optimizer in a future update, just like you suggest!! Cheers

bhack commented 3 years ago

You can follow also https://github.com/tensorflow/tensorflow/issues/48167

brianwa84 commented 3 years ago

That blog post is a great resource for how to glue the functional form in TFP together with tf.Variables (keras). I think we should probably fix the requirement that the parameters be a single 1D tensor, as this is inconsistent with other places in TFP, but it is not currently a high priority.