search

juntang-zhuang / Adabelief-Optimizer

Repository for NeurIPS 2020 Spotlight "AdaBelief Optimizer: Adapting stepsizes by the belief in observed gradients"
BSD 2-Clause "Simplified" License
1.05k stars 109 forks source link

issues on AdaBlief-tensorflow #27

Closed dusk666 closed 3 years ago

dusk666 commented 3 years ago

HI! I had some trouble using Adambelief in a simple lstm training. What could be the reason for this？ CODE： from adabelief_tf import AdaBeliefOptimizer tf.keras.backend.clear_session() multivariate_lstmA = tf.keras.models.Sequential([ LSTM(100, input_shape=input_shape, return_sequences=True), Flatten(), Dense(200, activation='relu'), Dropout(0.1), Dense(1) ]) model_checkpoint = tf.keras.callbacks.ModelCheckpoint( 'multivariate_lstmA.h5', monitor=('val_loss'), save_best_only=True) optimizer = AdaBeliefOptimizer(learning_rate=1e-3, epsilon=1e-14, rectify=False) multivariate_lstmA.compile(loss=loss, optimizer=optimizer, metrics=metric)

RESULT： Please check your arguments if you have upgraded adabelief-tf from version 0.0.1. Modifications to default arguments: eps weight_decouple rectify

adabelief-tf=0.0.1 1e-08 Not supported Not supported Current version (0.1.0) 1e-14 supported default: True For a complete table of recommended hyperparameters, see https://github.com/juntang-zhuang/Adabelief-Optimizer

dusk666 commented 3 years ago

python version：3.7.6 tensoflow version：tensorflow-gpu 2.2.0 adabelief-tf ：0.1.0 model using tf.keras.api

louislbc commented 3 years ago

Hi, does the adabelief-tf support tensorflow 1, for instance, tensorflow-gpu 1.14? I found the adabelief-tf uninstalled my tensorflow 1 and installed tensorflow 2. Thanks in advance.

dusk666 commented 3 years ago

Hi, does the adabelief-tf support tensorflow 1, for instance, tensorflow-gpu 1.14? I found the adabelief-tf uninstalled my tensorflow 1 and installed tensorflow 2. Thanks in advance.

I think u need tensorflow version>=2.1.0 , because neither adabelief-tf 0.0.1 nor 0.1.0 work in my tensorflow 2.0.0 env. BUT we can run it in tensorflow 2.2 or 2.3.

cryu854 commented 3 years ago

@dusk666 The message is to guide you to set the default arguments for AdaBelief, popping up once you create AdaBeliefOptimizer.

For version, the newest pypi_package cache file of adabelief_tf0.1.0 is not update yet. Currently, please use the following commands to install:

git clone https://github.com/juntang-zhuang/Adabelief-Optimizer.git
pip install ./Adabelief-Optimizer/pypi_packages/adabelief_tf0.1.0
cryu854 commented 3 years ago

@louislbc Sorry, the package currently only supports tensorflow 2.

juntang-zhuang commented 3 years ago

@louislbc @dusk666 We have released version 0.2.0, please try the new version by pip install adabelief-tf==0.2.0, this version should be slightly faster than 0.1.0.

juntang-zhuang commented 3 years ago

@louislbc Please see https://github.com/juntang-zhuang/Adabelief-Optimizer/tree/update_0.3.0/pypi_packages/adabelief_tf0.3.0, there are two subfolders, one for tf1 the other for tf2. It's not pushed to pip yet, so please download the source and try. Thanks for the implementation by @cryu854.