Open chenxinhua opened 3 years ago
I'm not sure since I'm not maintaining the tf version. @cryu854 is the main contributor for tf version. @cryu854 will you be available to implement a version for tf 1? If so we can merge it in release 0.3.0. Thanks a lot in advance.
It is difficult to get optimizer compatible with tf 1.x and tf 2.x at the same time, since tf.train.Optimizer
is deprecated in tf 2.x, but I will try to implement a tf1.x's version of Adabelief and notify you when I have any progress.
@cryu854 Thanks a lot!
@chenxinhua Please see https://github.com/juntang-zhuang/Adabelief-Optimizer/tree/update_0.3.0/pypi_packages/adabelief_tf0.3.0, there are two subfolders, one for tf1 the other for tf2. It's not pushed to pip yet, so please download the source and try. Thanks for the implementation by @cryu854.
Hey, @juntang-zhuang @cryu854 Seems the install script of the latest tf1 package pulls tf2: https://github.com/juntang-zhuang/Adabelief-Optimizer/blob/update_0.3.0/pypi_packages/adabelief_tf0.3.0/adabelief_tf1/setup.py#L22
I tried install it and verified the tf1 test script does not run with tf2 (2.4.1 which installed by the above ^ line)
I also tried tensorflow-gpu==1.15.5, the tf1 test script runs but throws the following error. Wonder if you have any insights.
ERROR: testSparseDevicePlacement (__main__.AdaBeliefOptimizerTest)
AdaBeliefOptimizerTest.testSparseDevicePlacement
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1348, in _run_fn
self._extend_graph()
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1388, in _extend_graph
tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Variable/IsInitialized/VarIsInitializedOp: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
AssignSubVariableOp: GPU CPU XLA_CPU XLA_GPU
Merge: GPU CPU XLA_CPU XLA_GPU
Identity: GPU CPU XLA_CPU XLA_GPU
GreaterEqual: GPU CPU XLA_CPU XLA_GPU
Square: GPU CPU XLA_CPU XLA_GPU
Switch: GPU CPU XLA_CPU XLA_GPU
Sqrt: GPU CPU XLA_CPU XLA_GPU
ResourceGather: GPU CPU XLA_CPU XLA_GPU
Cast: GPU CPU XLA_CPU XLA_GPU
ReadVariableOp: GPU CPU XLA_CPU XLA_GPU
AddV2: GPU CPU XLA_CPU XLA_GPU
Unique: GPU CPU
Shape: GPU CPU XLA_CPU XLA_GPU
AssignVariableOp: CPU XLA_CPU XLA_GPU
StridedSlice: GPU CPU XLA_CPU XLA_GPU
Sub: GPU CPU XLA_CPU XLA_GPU
Const: GPU CPU XLA_CPU XLA_GPU
VarIsInitializedOp: GPU CPU XLA_CPU XLA_GPU
VarHandleOp: GPU CPU XLA_CPU XLA_GPU
UnsortedSegmentSum: GPU CPU XLA_CPU XLA_GPU
NoOp: GPU CPU XLA_CPU XLA_GPU
GatherV2: GPU CPU XLA_CPU XLA_GPU
Mul: GPU CPU XLA_CPU XLA_GPU
ResourceScatterAdd: GPU CPU XLA_CPU XLA_GPU
RealDiv: GPU CPU XLA_CPU XLA_GPU
Colocation members, user-requested devices, and framework assigned devices, if any:
Variable/Initializer/initial_value (Const)
Variable (VarHandleOp) /device:GPU:0
Variable/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
Variable/Assign (AssignVariableOp) /device:GPU:0
Variable/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
Gather (ResourceGather) /device:GPU:0
beta1_power/Initializer/initial_value (Const) /device:GPU:0
beta1_power (VarHandleOp) /device:GPU:0
beta1_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
beta1_power/Assign (AssignVariableOp) /device:GPU:0
beta1_power/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
beta2_power/Initializer/initial_value (Const) /device:GPU:0
beta2_power (VarHandleOp) /device:GPU:0
beta2_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
beta2_power/Assign (AssignVariableOp) /device:GPU:0
beta2_power/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
step/Initializer/initial_value (Const) /device:GPU:0
step (VarHandleOp) /device:GPU:0
step/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
step/Assign (AssignVariableOp) /device:GPU:0
step/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
Variable/AdaBeliefOptimizer/Initializer/zeros (Const) /device:GPU:0
Variable/AdaBeliefOptimizer (VarHandleOp) /device:GPU:0
Variable/AdaBeliefOptimizer/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
Variable/AdaBeliefOptimizer/Assign (AssignVariableOp) /device:GPU:0
Variable/AdaBeliefOptimizer/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
Variable/AdaBeliefOptimizer_1/Initializer/zeros (Const) /device:GPU:0
Variable/AdaBeliefOptimizer_1 (VarHandleOp) /device:GPU:0
Variable/AdaBeliefOptimizer_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
Variable/AdaBeliefOptimizer_1/Assign (AssignVariableOp) /device:GPU:0
Variable/AdaBeliefOptimizer_1/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Unique (Unique) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Shape (Shape) /device:GPU:0
AdaBeliefOptimizer/update_Variable/strided_slice/stack (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/strided_slice/stack_1 (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/strided_slice/stack_2 (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/strided_slice (StridedSlice) /device:GPU:0
AdaBeliefOptimizer/update_Variable/UnsortedSegmentSum (UnsortedSegmentSum) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Cast/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Cast (Cast) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_1/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_1 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/AssignVariableOp (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ResourceScatterAdd (ResourceScatterAdd) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_1 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_2 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_1/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_1 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/GatherV2/axis (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/GatherV2 (GatherV2) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_2/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_2 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_3 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Square (Square) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_2 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_3 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_3 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/AssignVariableOp_1 (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_4 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/add (AddV2) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ResourceScatterAdd_1 (ResourceScatterAdd) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_5 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_6 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_4/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_4 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_1 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Sqrt (Sqrt) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_5/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_5 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_2/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_2 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_6/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_6 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_4/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_4 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_5/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_5 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_7 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_7/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_7 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_3 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_8 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_9/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_9 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_10/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_10 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_4 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_11/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_11 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_6 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_12/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_12 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_5 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_7 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_6 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Sqrt_1 (Sqrt) /device:GPU:0
AdaBeliefOptimizer/update_Variable/GreaterEqual (GreaterEqual) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/Switch (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/switch_t (Identity) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/switch_f (Identity) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/pred_id (Identity) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/mul/Switch (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/mul/Switch_1 (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/mul (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/add/Switch (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/add (AddV2) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/truediv (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/Switch_1 (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/Merge (Merge) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_8 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/AssignSubVariableOp (AssignSubVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_8 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/group_deps (NoOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/mul (Mul) /device:GPU:0
AdaBeliefOptimizer/AssignVariableOp (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_1 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_2 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/mul_1 (Mul) /device:GPU:0
AdaBeliefOptimizer/AssignVariableOp_1 (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_3 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_4 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/add/y (Const) /device:GPU:0
AdaBeliefOptimizer/add (AddV2) /device:GPU:0
AdaBeliefOptimizer/AssignVariableOp_2 (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_5 (ReadVariableOp) /device:GPU:0
Op: VarIsInitializedOp
Node attrs:
Registered kernels:
device='XLA_CPU'
device='XLA_GPU_JIT'
device='GPU'
device='CPU'
device='XLA_CPU_JIT'
device='XLA_GPU'
[[{{node Variable/IsInitialized/VarIsInitializedOp}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/test_util.py", line 1216, in decorated
return f(self, *args, **kwargs)
File "AdaBelief_tf1_test.py", line 141, in testSparseDevicePlacement
variables.global_variables_initializer().run()
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 2439, in run
_run_using_default_session(self, feed_dict, self.graph, session)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 5442, in _run_using_default_session
session.run(operation, feed_dict)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/test_util.py", line 1557, in run
return super(ErrorLoggingSession, self).run(*args, **kwargs)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Variable/IsInitialized/VarIsInitializedOp: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
AssignSubVariableOp: GPU CPU XLA_CPU XLA_GPU
Merge: GPU CPU XLA_CPU XLA_GPU
Identity: GPU CPU XLA_CPU XLA_GPU
GreaterEqual: GPU CPU XLA_CPU XLA_GPU
Square: GPU CPU XLA_CPU XLA_GPU
Switch: GPU CPU XLA_CPU XLA_GPU
Sqrt: GPU CPU XLA_CPU XLA_GPU
ResourceGather: GPU CPU XLA_CPU XLA_GPU
Cast: GPU CPU XLA_CPU XLA_GPU
ReadVariableOp: GPU CPU XLA_CPU XLA_GPU
AddV2: GPU CPU XLA_CPU XLA_GPU
Unique: GPU CPU
Shape: GPU CPU XLA_CPU XLA_GPU
AssignVariableOp: CPU XLA_CPU XLA_GPU
StridedSlice: GPU CPU XLA_CPU XLA_GPU
Sub: GPU CPU XLA_CPU XLA_GPU
Const: GPU CPU XLA_CPU XLA_GPU
VarIsInitializedOp: GPU CPU XLA_CPU XLA_GPU
VarHandleOp: GPU CPU XLA_CPU XLA_GPU
UnsortedSegmentSum: GPU CPU XLA_CPU XLA_GPU
NoOp: GPU CPU XLA_CPU XLA_GPU
GatherV2: GPU CPU XLA_CPU XLA_GPU
Mul: GPU CPU XLA_CPU XLA_GPU
ResourceScatterAdd: GPU CPU XLA_CPU XLA_GPU
RealDiv: GPU CPU XLA_CPU XLA_GPU
Colocation members, user-requested devices, and framework assigned devices, if any:
Variable/Initializer/initial_value (Const)
Variable (VarHandleOp) /device:GPU:0
Variable/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
Variable/Assign (AssignVariableOp) /device:GPU:0
Variable/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
Gather (ResourceGather) /device:GPU:0
beta1_power/Initializer/initial_value (Const) /device:GPU:0
beta1_power (VarHandleOp) /device:GPU:0
beta1_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
beta1_power/Assign (AssignVariableOp) /device:GPU:0
beta1_power/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
beta2_power/Initializer/initial_value (Const) /device:GPU:0
beta2_power (VarHandleOp) /device:GPU:0
beta2_power/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
beta2_power/Assign (AssignVariableOp) /device:GPU:0
beta2_power/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
step/Initializer/initial_value (Const) /device:GPU:0
step (VarHandleOp) /device:GPU:0
step/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
step/Assign (AssignVariableOp) /device:GPU:0
step/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
Variable/AdaBeliefOptimizer/Initializer/zeros (Const) /device:GPU:0
Variable/AdaBeliefOptimizer (VarHandleOp) /device:GPU:0
Variable/AdaBeliefOptimizer/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
Variable/AdaBeliefOptimizer/Assign (AssignVariableOp) /device:GPU:0
Variable/AdaBeliefOptimizer/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
Variable/AdaBeliefOptimizer_1/Initializer/zeros (Const) /device:GPU:0
Variable/AdaBeliefOptimizer_1 (VarHandleOp) /device:GPU:0
Variable/AdaBeliefOptimizer_1/IsInitialized/VarIsInitializedOp (VarIsInitializedOp) /device:GPU:0
Variable/AdaBeliefOptimizer_1/Assign (AssignVariableOp) /device:GPU:0
Variable/AdaBeliefOptimizer_1/Read/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Unique (Unique) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Shape (Shape) /device:GPU:0
AdaBeliefOptimizer/update_Variable/strided_slice/stack (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/strided_slice/stack_1 (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/strided_slice/stack_2 (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/strided_slice (StridedSlice) /device:GPU:0
AdaBeliefOptimizer/update_Variable/UnsortedSegmentSum (UnsortedSegmentSum) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Cast/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Cast (Cast) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_1/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_1 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/AssignVariableOp (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ResourceScatterAdd (ResourceScatterAdd) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_1 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_2 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_1/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_1 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/GatherV2/axis (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/GatherV2 (GatherV2) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_2/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_2 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_3 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Square (Square) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_2 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_3 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_3 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/AssignVariableOp_1 (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_4 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/add (AddV2) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ResourceScatterAdd_1 (ResourceScatterAdd) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_5 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_6 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_4/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_4 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_1 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Sqrt (Sqrt) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_5/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_5 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_2/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_2 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_6/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_6 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_4/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_4 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_5/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_5 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_7 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_7/x (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_7 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_3 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_8 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_9/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_9 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_10/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_10 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_4 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_11/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_11 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_6 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_12/y (Const) /device:GPU:0
AdaBeliefOptimizer/update_Variable/sub_12 (Sub) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_5 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_7 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/truediv_6 (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/Sqrt_1 (Sqrt) /device:GPU:0
AdaBeliefOptimizer/update_Variable/GreaterEqual (GreaterEqual) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/Switch (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/switch_t (Identity) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/switch_f (Identity) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/pred_id (Identity) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/mul/Switch (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/mul/Switch_1 (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/mul (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/add/Switch (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/add (AddV2) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/truediv (RealDiv) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/Switch_1 (Switch) /device:GPU:0
AdaBeliefOptimizer/update_Variable/cond/Merge (Merge) /device:GPU:0
AdaBeliefOptimizer/update_Variable/mul_8 (Mul) /device:GPU:0
AdaBeliefOptimizer/update_Variable/AssignSubVariableOp (AssignSubVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/ReadVariableOp_8 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/update_Variable/group_deps (NoOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/mul (Mul) /device:GPU:0
AdaBeliefOptimizer/AssignVariableOp (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_1 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_2 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/mul_1 (Mul) /device:GPU:0
AdaBeliefOptimizer/AssignVariableOp_1 (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_3 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_4 (ReadVariableOp) /device:GPU:0
AdaBeliefOptimizer/add/y (Const) /device:GPU:0
AdaBeliefOptimizer/add (AddV2) /device:GPU:0
AdaBeliefOptimizer/AssignVariableOp_2 (AssignVariableOp) /device:GPU:0
AdaBeliefOptimizer/ReadVariableOp_5 (ReadVariableOp) /device:GPU:0
Op: VarIsInitializedOp
Node attrs:
Registered kernels:
device='XLA_CPU'
device='XLA_GPU_JIT'
device='GPU'
device='CPU'
device='XLA_CPU_JIT'
device='XLA_GPU'
[[node Variable/IsInitialized/VarIsInitializedOp (defined at /home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
Original stack trace for 'Variable/IsInitialized/VarIsInitializedOp':
File "AdaBelief_tf1_test.py", line 574, in <module>
test.main()
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/test.py", line 64, in main
return _googletest.main(argv)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/googletest.py", line 65, in main
benchmark.benchmarks_main(true_main=main_wrapper)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/benchmark.py", line 407, in benchmarks_main
true_main()
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/googletest.py", line 64, in main_wrapper
return app.run(main=g_main, argv=args)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/app.py", line 303, in run
_run_main(main, args)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/platform/googletest.py", line 55, in g_main
absltest_main(argv=argv)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/absltest.py", line 2002, in main
_run_in_app(run_tests, args, kwargs)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/absltest.py", line 2109, in _run_in_app
function(argv, args, kwargs)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/absltest.py", line 2434, in run_tests
argv, args, kwargs, xml_reporter.TextAndXMLTestRunner)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/absltest.py", line 2404, in _run_and_get_tests_result
test_program = unittest.TestProgram(*args, **kwargs)
File "/usr/lib/python3.6/unittest/main.py", line 95, in __init__
self.runTests()
File "/usr/lib/python3.6/unittest/main.py", line 256, in runTests
self.result = testRunner.run(self.test)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/absl_py-0.11.0-py3.6.egg/absl/testing/_pretty_print_reporter.py", line 87, in run
return super(TextTestRunner, self).run(test)
File "/usr/lib/python3.6/unittest/runner.py", line 176, in run
test(result)
File "/usr/lib/python3.6/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.6/unittest/suite.py", line 122, in run
test(result)
File "/usr/lib/python3.6/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.6/unittest/suite.py", line 122, in run
test(result)
File "/usr/lib/python3.6/unittest/case.py", line 653, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.6/unittest/case.py", line 605, in run
testMethod()
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/test_util.py", line 1216, in decorated
return f(self, *args, **kwargs)
File "AdaBelief_tf1_test.py", line 136, in testSparseDevicePlacement
var = variables.Variable([[1.0], [2.0]])
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 260, in __call__
return cls._variable_v2_call(*args, **kwargs)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 254, in _variable_v2_call
shape=shape)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 235, in <lambda>
previous_getter = lambda **kws: default_variable_creator_v2(None, **kws)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variable_scope.py", line 2552, in default_variable_creator_v2
shape=shape)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 262, in __call__
return super(VariableMetaclass, cls).__call__(*args, **kwargs)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1406, in __init__
distribute_strategy=distribute_strategy)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1567, in _init_from_args
gen_resource_variable_ops.var_is_initialized_op(handle))
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_resource_variable_ops.py", line 1504, in var_is_initialized_op
"VarIsInitializedOp", resource=resource, name=name)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/home/ubuntu/venv-adabelief-tf1/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()
Any thoughts on this @juntang-zhuang @cryu854 ? Thanks!
Sorry for the late reply. Please use tensorflow 2 or use the cpu training for now, I will start investigating this issue soon and note you once I have any progress.
Sorry for the late reply. Please use tensorflow 2 or use the cpu for trainingt for now, I will start investigating this issue soon and note you once I have any progress.
Thank you for looking into this. Look forward to the fix!
will adabelief support for tensorflow 1.10+