search

juntang-zhuang / Adabelief-Optimizer

Repository for NeurIPS 2020 Spotlight "AdaBelief Optimizer: Adapting stepsizes by the belief in observed gradients"
BSD 2-Clause "Simplified" License
1.05k stars 108 forks source link

support for tensorflow 1.10+ #37

Open chenxinhua opened 3 years ago

chenxinhua commented 3 years ago

will adabelief support for tensorflow 1.10+

juntang-zhuang commented 3 years ago

I'm not sure since I'm not maintaining the tf version. @cryu854 is the main contributor for tf version. @cryu854 will you be available to implement a version for tf 1? If so we can merge it in release 0.3.0. Thanks a lot in advance.

cryu854 commented 3 years ago

It is difficult to get optimizer compatible with tf 1.x and tf 2.x at the same time, since tf.train.Optimizer is deprecated in tf 2.x, but I will try to implement a tf1.x's version of Adabelief and notify you when I have any progress.

juntang-zhuang commented 3 years ago

@cryu854 Thanks a lot!

juntang-zhuang commented 3 years ago

@chenxinhua Please see https://github.com/juntang-zhuang/Adabelief-Optimizer/tree/update_0.3.0/pypi_packages/adabelief_tf0.3.0, there are two subfolders, one for tf1 the other for tf2. It's not pushed to pip yet, so please download the source and try. Thanks for the implementation by @cryu854.

chuanli11 commented 3 years ago

Hey, @juntang-zhuang @cryu854 Seems the install script of the latest tf1 package pulls tf2: https://github.com/juntang-zhuang/Adabelief-Optimizer/blob/update_0.3.0/pypi_packages/adabelief_tf0.3.0/adabelief_tf1/setup.py#L22

I tried install it and verified the tf1 test script does not run with tf2 (2.4.1 which installed by the above ^ line)

I also tried tensorflow-gpu==1.15.5, the tf1 test script runs but throws the following error. Wonder if you have any insights. 

ERROR: testSparseDevicePlacement (__main__.AdaBeliefOptimizerTest)
AdaBeliefOptimizerTest.testSparseDevicePlacement
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1348, in _run_fn
    self._extend_graph()
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1388, in _extend_graph
    tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Variable/IsInitialized/VarIsInitializedOp: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
AssignSubVariableOp: GPU CPU XLA_CPU XLA_GPU 
Merge: GPU CPU XLA_CPU XLA_GPU 
Identity: GPU CPU XLA_CPU XLA_GPU 
GreaterEqual: GPU CPU XLA_CPU XLA_GPU 
Square: GPU CPU XLA_CPU XLA_GPU 
Switch: GPU CPU XLA_CPU XLA_GPU 
Sqrt: GPU CPU XLA_CPU XLA_GPU 
ResourceGather: GPU CPU XLA_CPU XLA_GPU 
Cast: GPU CPU XLA_CPU XLA_GPU 
ReadVariableOp: GPU CPU XLA_CPU XLA_GPU 
AddV2: GPU CPU XLA_CPU XLA_GPU 
Unique: GPU CPU 
Shape: GPU CPU XLA_CPU XLA_GPU 
AssignVariableOp: CPU XLA_CPU XLA_GPU 
StridedSlice: GPU CPU XLA_CPU XLA_GPU 
Sub: GPU CPU XLA_CPU XLA_GPU 
Const: GPU CPU XLA_CPU XLA_GPU 
VarIsInitializedOp: GPU CPU XLA_CPU XLA_GPU 
VarHandleOp: GPU CPU XLA_CPU XLA_GPU 
UnsortedSegmentSum: GPU CPU XLA_CPU XLA_GPU 
NoOp: GPU CPU XLA_CPU XLA_GPU 
GatherV2: GPU CPU XLA_CPU XLA_GPU 
Mul: GPU CPU XLA_CPU XLA_GPU 
ResourceScatterAdd: GPU CPU XLA_CPU XLA_GPU 
RealDiv: GPU CPU XLA_CPU XLA_GPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  Variable/Initializer/initial_value (Const) 
  Variable (VarHandleOp) /device:GPU:0
  Variable/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  Variable/Assign (AssignVariableOp) /device:GPU:0
  Variable/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  Gather (ResourceGather) /device:GPU:0
  beta1_power/Initializer/initial_value (Const) /device:GPU:0
  beta1_power (VarHandleOp) /device:GPU:0
  beta1_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  beta1_power/Assign (AssignVariableOp) /device:GPU:0
  beta1_power/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  beta2_power/Initializer/initial_value (Const) /device:GPU:0
  beta2_power (VarHandleOp) /device:GPU:0
  beta2_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  beta2_power/Assign (AssignVariableOp) /device:GPU:0
  beta2_power/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  step/Initializer/initial_value (Const) /device:GPU:0
  step (VarHandleOp) /device:GPU:0
  step/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  step/Assign (AssignVariableOp) /device:GPU:0
  step/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  Variable/AdaBeliefOptimizer/Initializer/zeros (Const) /device:GPU:0
  Variable/AdaBeliefOptimizer (VarHandleOp) /device:GPU:0
  Variable/AdaBeliefOptimizer/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  Variable/AdaBeliefOptimizer/Assign (AssignVariableOp) /device:GPU:0
  Variable/AdaBeliefOptimizer/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  Variable/AdaBeliefOptimizer_1/Initializer/zeros (Const) /device:GPU:0
  Variable/AdaBeliefOptimizer_1 (VarHandleOp) /device:GPU:0
  Variable/AdaBeliefOptimizer_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  Variable/AdaBeliefOptimizer_1/Assign (AssignVariableOp) /device:GPU:0
  Variable/AdaBeliefOptimizer_1/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Unique (Unique) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Shape (Shape) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/strided_slice/stack (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/strided_slice/stack_1 (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/strided_slice/stack_2 (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/strided_slice (StridedSlice) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/UnsortedSegmentSum (UnsortedSegmentSum) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Cast/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Cast (Cast) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_1/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_1 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/AssignVariableOp (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ResourceScatterAdd (ResourceScatterAdd) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_1 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_2 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_1/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_1 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/GatherV2/axis (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/GatherV2 (GatherV2) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_2/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_2 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_3 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Square (Square) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_2 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_3 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_3 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/AssignVariableOp_1 (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_4 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/add (AddV2) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ResourceScatterAdd_1 (ResourceScatterAdd) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_5 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_6 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_4/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_4 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_1 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Sqrt (Sqrt) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_5/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_5 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_2/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_2 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_6/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_6 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_4/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_4 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_5/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_5 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_7 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_7/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_7 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_3 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_8 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_9/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_9 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_10/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_10 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_4 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_11/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_11 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_6 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_12/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_12 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_5 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_7 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_6 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Sqrt_1 (Sqrt) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/GreaterEqual (GreaterEqual) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/Switch (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/switch_t (Identity) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/switch_f (Identity) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/pred_id (Identity) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/mul/Switch (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/mul/Switch_1 (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/mul (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/add/Switch (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/add (AddV2) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/truediv (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/Switch_1 (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/Merge (Merge) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_8 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/AssignSubVariableOp (AssignSubVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_8 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/group_deps (NoOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/mul (Mul) /device:GPU:0
  AdaBeliefOptimizer/AssignVariableOp (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_1 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_2 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/mul_1 (Mul) /device:GPU:0
  AdaBeliefOptimizer/AssignVariableOp_1 (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_3 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_4 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/add/y (Const) /device:GPU:0
  AdaBeliefOptimizer/add (AddV2) /device:GPU:0
  AdaBeliefOptimizer/AssignVariableOp_2 (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_5 (ReadVariableOp) /device:GPU:0

Op: VarIsInitializedOp
Node attrs: 
Registered kernels:
  device='XLA_CPU'
  device='XLA_GPU_JIT'
  device='GPU'
  device='CPU'
  device='XLA_CPU_JIT'
  device='XLA_GPU'

     [[{{node Variable/IsInitialized/VarIsInitializedOp}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/test_util.py", line 1216, in decorated
    return f(self, *args, **kwargs)
  File "AdaBelief_tf1_test.py", line 141, in testSparseDevicePlacement
    variables.global_variables_initializer().run()
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 2439, in run
    _run_using_default_session(self, feed_dict, self.graph, session)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 5442, in _run_using_default_session
    session.run(operation, feed_dict)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/test_util.py", line 1557, in run
    return super(ErrorLoggingSession, self).run(*args, **kwargs)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Variable/IsInitialized/VarIsInitializedOp: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices: 
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
AssignSubVariableOp: GPU CPU XLA_CPU XLA_GPU 
Merge: GPU CPU XLA_CPU XLA_GPU 
Identity: GPU CPU XLA_CPU XLA_GPU 
GreaterEqual: GPU CPU XLA_CPU XLA_GPU 
Square: GPU CPU XLA_CPU XLA_GPU 
Switch: GPU CPU XLA_CPU XLA_GPU 
Sqrt: GPU CPU XLA_CPU XLA_GPU 
ResourceGather: GPU CPU XLA_CPU XLA_GPU 
Cast: GPU CPU XLA_CPU XLA_GPU 
ReadVariableOp: GPU CPU XLA_CPU XLA_GPU 
AddV2: GPU CPU XLA_CPU XLA_GPU 
Unique: GPU CPU 
Shape: GPU CPU XLA_CPU XLA_GPU 
AssignVariableOp: CPU XLA_CPU XLA_GPU 
StridedSlice: GPU CPU XLA_CPU XLA_GPU 
Sub: GPU CPU XLA_CPU XLA_GPU 
Const: GPU CPU XLA_CPU XLA_GPU 
VarIsInitializedOp: GPU CPU XLA_CPU XLA_GPU 
VarHandleOp: GPU CPU XLA_CPU XLA_GPU 
UnsortedSegmentSum: GPU CPU XLA_CPU XLA_GPU 
NoOp: GPU CPU XLA_CPU XLA_GPU 
GatherV2: GPU CPU XLA_CPU XLA_GPU 
Mul: GPU CPU XLA_CPU XLA_GPU 
ResourceScatterAdd: GPU CPU XLA_CPU XLA_GPU 
RealDiv: GPU CPU XLA_CPU XLA_GPU 

Colocation members, user-requested devices, and framework assigned devices, if any:
  Variable/Initializer/initial_value (Const) 
  Variable (VarHandleOp) /device:GPU:0
  Variable/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  Variable/Assign (AssignVariableOp) /device:GPU:0
  Variable/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  Gather (ResourceGather) /device:GPU:0
  beta1_power/Initializer/initial_value (Const) /device:GPU:0
  beta1_power (VarHandleOp) /device:GPU:0
  beta1_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  beta1_power/Assign (AssignVariableOp) /device:GPU:0
  beta1_power/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  beta2_power/Initializer/initial_value (Const) /device:GPU:0
  beta2_power (VarHandleOp) /device:GPU:0
  beta2_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  beta2_power/Assign (AssignVariableOp) /device:GPU:0
  beta2_power/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  step/Initializer/initial_value (Const) /device:GPU:0
  step (VarHandleOp) /device:GPU:0
  step/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  step/Assign (AssignVariableOp) /device:GPU:0
  step/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  Variable/AdaBeliefOptimizer/Initializer/zeros (Const) /device:GPU:0
  Variable/AdaBeliefOptimizer (VarHandleOp) /device:GPU:0
  Variable/AdaBeliefOptimizer/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  Variable/AdaBeliefOptimizer/Assign (AssignVariableOp) /device:GPU:0
  Variable/AdaBeliefOptimizer/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  Variable/AdaBeliefOptimizer_1/Initializer/zeros (Const) /device:GPU:0
  Variable/AdaBeliefOptimizer_1 (VarHandleOp) /device:GPU:0
  Variable/AdaBeliefOptimizer_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
  Variable/AdaBeliefOptimizer_1/Assign (AssignVariableOp) /device:GPU:0
  Variable/AdaBeliefOptimizer_1/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Unique (Unique) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Shape (Shape) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/strided_slice/stack (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/strided_slice/stack_1 (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/strided_slice/stack_2 (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/strided_slice (StridedSlice) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/UnsortedSegmentSum (UnsortedSegmentSum) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Cast/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Cast (Cast) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_1/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_1 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/AssignVariableOp (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ResourceScatterAdd (ResourceScatterAdd) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_1 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_2 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_1/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_1 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/GatherV2/axis (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/GatherV2 (GatherV2) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_2/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_2 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_3 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Square (Square) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_2 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_3 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_3 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/AssignVariableOp_1 (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_4 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/add (AddV2) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ResourceScatterAdd_1 (ResourceScatterAdd) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_5 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_6 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_4/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_4 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_1 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Sqrt (Sqrt) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_5/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_5 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_2/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_2 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_6/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_6 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_4/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_4 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_5/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_5 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_7 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_7/x (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_7 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_3 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_8 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_9/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_9 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_10/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_10 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_4 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_11/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_11 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_6 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_12/y (Const) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/sub_12 (Sub) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_5 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_7 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/truediv_6 (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/Sqrt_1 (Sqrt) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/GreaterEqual (GreaterEqual) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/Switch (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/switch_t (Identity) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/switch_f (Identity) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/pred_id (Identity) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/mul/Switch (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/mul/Switch_1 (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/mul (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/add/Switch (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/add (AddV2) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/truediv (RealDiv) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/Switch_1 (Switch) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/cond/Merge (Merge) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/mul_8 (Mul) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/AssignSubVariableOp (AssignSubVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/ReadVariableOp_8 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/update_Variable/group_deps (NoOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/mul (Mul) /device:GPU:0
  AdaBeliefOptimizer/AssignVariableOp (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_1 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_2 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/mul_1 (Mul) /device:GPU:0
  AdaBeliefOptimizer/AssignVariableOp_1 (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_3 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_4 (ReadVariableOp) /device:GPU:0
  AdaBeliefOptimizer/add/y (Const) /device:GPU:0
  AdaBeliefOptimizer/add (AddV2) /device:GPU:0
  AdaBeliefOptimizer/AssignVariableOp_2 (AssignVariableOp) /device:GPU:0
  AdaBeliefOptimizer/ReadVariableOp_5 (ReadVariableOp) /device:GPU:0

Op: VarIsInitializedOp
Node attrs: 
Registered kernels:
  device='XLA_CPU'
  device='XLA_GPU_JIT'
  device='GPU'
  device='CPU'
  device='XLA_CPU_JIT'
  device='XLA_GPU'

     [[node Variable/IsInitialized/VarIsInitializedOp (defined at /home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'Variable/IsInitialized/VarIsInitializedOp':
  File "AdaBelief_tf1_test.py", line 574, in <module>
    test.main()
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/test.py", line 64, in main
    return _googletest.main(argv)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/googletest.py", line 65, in main
    benchmark.benchmarks_main(true_main=main_wrapper)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/benchmark.py", line 407, in benchmarks_main
    true_main()
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/googletest.py", line 64, in main_wrapper
    return app.run(main=g_main, argv=args)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/googletest.py", line 55, in g_main
    absltest_main(argv=argv)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/absltest.py", line 2002, in main
    _run_in_app(run_tests, args, kwargs)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/absltest.py", line 2109, in _run_in_app
    function(argv, args, kwargs)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/absltest.py", line 2434, in run_tests
    argv, args, kwargs, xml_reporter.TextAndXMLTestRunner)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/absltest.py", line 2404, in _run_and_get_tests_result
    test_program = unittest.TestProgram(*args, **kwargs)
  File "/usr/lib/python3.6/unittest/main.py", line 95, in __init__
    self.runTests()
  File "/usr/lib/python3.6/unittest/main.py", line 256, in runTests
    self.result = testRunner.run(self.test)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/_pretty_print_reporter.py", line 87, in run
    return super(TextTestRunner, self).run(test)
  File "/usr/lib/python3.6/unittest/runner.py", line 176, in run
    test(result)
  File "/usr/lib/python3.6/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.6/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.6/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.6/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.6/unittest/case.py", line 653, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.6/unittest/case.py", line 605, in run
    testMethod()
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/test_util.py", line 1216, in decorated
    return f(self, *args, **kwargs)
  File "AdaBelief_tf1_test.py", line 136, in testSparseDevicePlacement
    var = variables.Variable([[1.0], [2.0]])
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 260, in __call__
    return cls._variable_v2_call(*args, **kwargs)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 254, in _variable_v2_call
    shape=shape)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 235, in <lambda>
    previous_getter = lambda **kws: default_variable_creator_v2(None, **kws)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variable_scope.py", line 2552, in default_variable_creator_v2
    shape=shape)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 262, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1406, in __init__
    distribute_strategy=distribute_strategy)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1567, in _init_from_args
    gen_resource_variable_ops.var_is_initialized_op(handle))
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_resource_variable_ops.py", line 1504, in var_is_initialized_op
    "VarIsInitializedOp", resource=resource, name=name)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()
chuanli11 commented 3 years ago

Any thoughts on this @juntang-zhuang @cryu854 ? Thanks!

cryu854 commented 3 years ago

Sorry for the late reply. Please use tensorflow 2 or use the cpu training for now, I will start investigating this issue soon and note you once I have any progress.

chuanli11 commented 3 years ago

Sorry for the late reply. Please use tensorflow 2 or use the cpu for trainingt for now, I will start investigating this issue soon and note you once I have any progress.

Thank you for looking into this. Look forward to the fix!