Usage of two optimizers in the fit function

[StillerPatrick]

Can you explain why do you use the LBFGS-Optimzer and the Adam-Optimzer in each training-step `

    for it in range(nIter):
        self.sess.run(self.train_op_Adam, tf_dict)

        # Print
        if it % 10 == 0:
            elapsed = time.time() - start_time
            loss_value = self.sess.run(self.loss, tf_dict)
            print('It: %d, Loss: %.3e, Time: %.2f' % 
                  (it, loss_value, elapsed))
            start_time = time.time()

        self.optimizer.minimize(self.sess, 
                                feed_dict = tf_dict,         
                                fetches = [self.loss], 
                                loss_callback = self.callback)

`

[lauraxinzhang]

It's been a couple month since @StillerPatrick posted this, but if you're still interested - Note how the LBFGS optimizer is not actually called within the for loop: ` def train(self, nIter): tf_dict = {self.x0_tf: self.x0, self.u0_tf: self.u0, self.x1_tf: self.x1, self.dummy_x0_tf: np.ones((self.x0.shape[0], self.q)), self.dummy_x1_tf: np.ones((self.x1.shape[0], self.q+1))}

    start_time = time.time()
    for it in range(nIter):
        self.sess.run(self.train_op_Adam, tf_dict)

        # Print
        if it % 10 == 0:
            elapsed = time.time() - start_time
            loss_value = self.sess.run(self.loss, tf_dict)
            print('It: %d, Loss: %.3e, Time: %.2f' % 
                  (it, loss_value, elapsed))
            start_time = time.time()
    # HERE notice the un-indent
    self.optimizer.minimize(self.sess,
                            feed_dict = tf_dict,
                            fetches = [self.loss],
                            loss_callback = self.callback)`

The max iterations and other parameters of the LBFGS are predefined in the class constructors.

So it seems to me, that when you run the code as-is, the model is actually first trained with adam and then continued to train with LBFGS. I suspect the authors tried to train the model with both and just left the code there...

[StillerPatrick]

Ah ok, thanks for your answer. In further works the author works only with LBFGS and sets the number of adam optimization steps to zero.

[yang12313]

I commented out the self.optimizer.minimize(self.sess, feed_dict = tf_dict, fetches = [self.loss], loss_callback = self.callback) and the code is still working like before. so I assumed the lbfgs is not actually working. only the adam works in the training session.