search

heaversm / deeplab-training

Training your own Deeplab Model in Tensorflow
37 stars 22 forks source link

Error when training with pretrained model #11

Open At-Walid opened 1 year ago

At-Walid commented 1 year ago

Hello,

I followed your tutorial but I'm getting the following error:

`ERROR:tensorflow:================================== Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>): <tf.Tensor 'init_ops/report_uninitialized_variables/boolean_mask/GatherV2:0' shape=(?,) dtype=string> If you want to mark it as used call its "mark_used()" method. It was originally created here: File "/home/users/mattaoui/.local/lib/python3.7/site-packages/absl/app.py", line 330, in run raise File "/home/users/mattaoui/.local/lib/python3.7/site-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "/home/users/mattaoui/models/research/deeplab/train.py", line 458, in main save_interval_secs=FLAGS.save_interval_secs) File "/home/users/mattaoui/.conda/envs/tf-venv/lib/python3.7/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 796, in train should_retry = True File "/home/users/mattaoui/.conda/envs/tf-venv/lib/python3.7/site-packages/tensorflow_core/python/util/tf_should_use.py", line 198, in wrapped return _add_should_use_warning(fn(*args, **kwargs))

E0417 15:02:19.151673 140737354008384 tf_should_use.py:76] ================================== Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>): <tf.Tensor 'init_ops/report_uninitialized_variables/boolean_mask/GatherV2:0' shape=(?,) dtype=string> If you want to mark it as used call its "mark_used()" method. It was originally created here: File "/home/users/mattaoui/.local/lib/python3.7/site-packages/absl/app.py", line 330, in run raise File "/home/users/mattaoui/.local/lib/python3.7/site-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "/home/users/mattaoui/models/research/deeplab/train.py", line 458, in main save_interval_secs=FLAGS.save_interval_secs) File "/home/users/mattaoui/.conda/envs/tf-venv/lib/python3.7/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 796, in train should_retry = True File "/home/users/mattaoui/.conda/envs/tf-venv/lib/python3.7/site-packages/tensorflow_core/python/util/tf_should_use.py", line 198, in wrapped return _add_should_use_warning(fn(*args, **kwargs)) ==================================`

heaversm commented 1 year ago

Looks like the SyncReplicaOptimizer is no longer supported by tensorflow. DistributedStrategy can be used to replace the functionalities. I found that out here and the solve is described in further detail here.

At-Walid commented 1 year ago

Thank you for your response. I looked at the code, and I can't find where SyncReplicaOptimizer is used