search

nicknochnack / TFODCourse

962 stars 923 forks source link

not training with new Tensorflow version!! please help!! #84

Open PrakharPrashantSingh opened 2 years ago

PrakharPrashantSingh commented 2 years ago

Skipping registering GPU devices... 2022-04-22 14:20:05.286806: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. WARNING:tensorflow:There are non-GPU devices in tf.distribute.Strategy, not using nccl allreduce. W0422 14:20:05.297503 34508 cross_device_ops.py:1387] There are non-GPU devices in tf.distribute.Strategy, not using nccl allreduce. INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0422 14:20:05.297503 34508 mirrored_strategy.py:369] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) INFO:tensorflow:Maybe overwriting train_steps: 2000 I0422 14:20:05.313093 34508 config_util.py:552] Maybe overwriting train_steps: 2000 INFO:tensorflow:Maybe overwriting use_bfloat16: False I0422 14:20:05.313093 34508 config_util.py:552] Maybe overwriting use_bfloat16: False WARNING:tensorflow:From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\model_lib_v2.py:564: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version. Instructions for updating: rename to distribute_datasets_from_function W0422 14:20:05.422441 34508 deprecation.py:345] From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\model_lib_v2.py:564: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version. Instructions for updating: rename to distribute_datasets_from_function INFO:tensorflow:Reading unweighted datasets: ['Tensorflow\workspace\annotations\train.record'] I0422 14:20:05.438063 34508 dataset_builder.py:162] Reading unweighted datasets: ['Tensorflow\workspace\annotations\train.record'] INFO:tensorflow:Reading record datasets for input file: ['Tensorflow\workspace\annotations\train.record'] I0422 14:20:05.438063 34508 dataset_builder.py:79] Reading record datasets for input file: ['Tensorflow\workspace\annotations\train.record'] INFO:tensorflow:Number of filenames to read: 1 I0422 14:20:05.438063 34508 dataset_builder.py:80] Number of filenames to read: 1 WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. W0422 14:20:05.438063 34508 dataset_builder.py:87] num_readers has been reduced to 1 to match input file shards. WARNING:tensorflow:From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\builders\dataset_builder.py:104: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic. W0422 14:20:05.438063 34508 deprecation.py:345] From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\builders\dataset_builder.py:104: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic. WARNING:tensorflow:From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map() W0422 14:20:05.484927 34508 deprecation.py:345] From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Usetf.data.Dataset.map() WARNING:tensorflow:From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\util\dispatch.py:206: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead. W0422 14:20:14.685878 34508 deprecation.py:345] From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\util\dispatch.py:206: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead. WARNING:tensorflow:From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\util\dispatch.py:206: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead. W0422 14:20:18.809943 34508 deprecation.py:345] From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\util\dispatch.py:206: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead. WARNING:tensorflow:From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\autograph\impl\api.py:464: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W0422 14:20:21.153142 34508 deprecation.py:345] From C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\autograph\impl\api.py:464: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. 2022-04-22 14:20:26.073242: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2) 2022-04-22 14:20:26.360368: W tensorflow/core/framework/dataset.cc:679] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. Traceback (most recent call last): File "Tensorflow\models\research\object_detection\model_main_tf2.py", line 114, in tf.compat.v1.app.run() File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\absl\app.py", line 312, in run _run_main(main, args) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\absl\app.py", line 258, in _run_main sys.exit(main(argv)) File "Tensorflow\models\research\object_detection\model_main_tf2.py", line 111, in main record_summaries=FLAGS.record_summaries) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\model_lib_v2.py", line 609, in train_loop train_input, unpad_groundtruth_tensors) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\model_lib_v2.py", line 400, in load_fine_tune_checkpoint _ensure_model_is_built(model, input_dataset, unpad_groundtruth_tensors) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\object_detection-0.1-py3.6.egg\object_detection\model_lib_v2.py", line 160, in _ensure_model_is_built features, labels = iter(input_dataset).next() File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 689, in next return self.next() File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 693, in next return self.get_next() File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 731, in get_next self._iterators[i].get_next_as_list_static_shapes(new_name)) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1951, in get_next_as_list_static_shapes return self._format_data_list_with_options(self._iterator.get_next()) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\data\ops\multi_device_iterator_ops.py", line 573, in get_next result.append(self._device_iterators[i].get_next()) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 814, in get_next return self._next_internal() File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 747, in _next_internal output_shapes=self._flat_output_shapes) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2727, in iterator_get_next _ops.raise_from_not_ok_status(e, name) File "C:\Users\psh4\Anaconda3\envs\prakhar\lib\site-packages\tensorflow\python\framework\ops.py", line 6941, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [[0.37890625]] [[0.216796875]] [[{{node Assert_1/AssertGuard/else/_35/Assert_1/AssertGuard/Assert}}]] [[MultiDeviceIteratorGetNextFromShard]] [[RemoteCall]] [Op:IteratorGetNext]

ghaithAlMasri commented 2 years ago

Same here