Why L-BFGS performs different in tensorflow.compat.v1 and pytorch

lululxvi / deepxde

A library for scientific machine learning and physics-informed learning

https://deepxde.readthedocs.io

GNU Lesser General Public License v2.1

2.63k stars 739 forks source link

Why L-BFGS performs different in tensorflow.compat.v1 and pytorch #1698

Open gaohuizhang opened 6 months ago

gaohuizhang commented 6 months ago

Hi,

I tried to train the model with L-BFGS after 15000 iterations of Adam, but I got different results from tensorflow.compat.v1 and PyTorch, even if I use the exact same code, just a different backend. This is the loss history using tensorflow.compat.v1 This is the loss history using Pytorch pytorch

Does anyone have any idea why this happens? Does this mean we have to use TensorFlow if we want to use the L-BFGS optimizer?

Thanks a lot

bakhtiyar-k commented 6 months ago

Hello, try to use the same seed for both cases dde.config.set_random_seed(1). Maybe the results are different due to different initialization

gaohuizhang commented 6 months ago

Thanks for your advice, but I tried several times and got similar results. I don't think it is caused by randomness of initialization.

praksharma commented 5 months ago

Because LBFGS is implemented differently in both libraries. You find the link the the particular implementation (papers) in the source code of tf and torch.