Invoking ptxas not supported on Windows

edwardHujber commented 5 years ago

System information

What is the top-level directory of the model you are using:
\models\research\object_detection\
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): NO, trying to use model_main.py
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): v1.15.0-rc2-10-g38ea9bbfea 1.15.0-rc3
Bazel version (if compiling from source): N/A
CUDA/cuDNN version: CUDA Version 10.0.130 cuDNN: 7.6.4.38
GPU model and memory: GeForce RTX 2080 SUPER. 8 GB dedicated, 32 GB shared
Exact command to reproduce: From within an Anaconda environment: python model_main.py --alsologtostderr --model_dir=training/trial_1/ --pipeline_config_path=training/trial_1/faster_rcnn_nas_coco.config

Describe the problem

Hangs on a

W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows
Relying on driver to perform ptx compilation. This message will be only logged once.

message. Sits there forever. Sometimes (usually after restarting the terminal and clearing out any produced files like .ckpt and .pbtxt ) it gets passed this point and soon after crashes with an out of memory problem. Mentioning that because I don't know if its related or not.

Source code / logs

(tensorflow) F:\Hujber\TensorFlow\workspace\wormLearn>python model_main.py --alsologtostderr --model_dir=training/trial_1/ --pipeline_config_path=training/trial_1/faster_rcnn_nas_coco.config
2019-10-09 23:43:04.866391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\slim\nets\inception_resnet_v2.py:373: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\slim\nets\mobilenet\mobilenet.py:389: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

WARNING:tensorflow:From model_main.py:109: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\utils\config_util.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W1009 23:43:07.285009 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\utils\config_util.py:94: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:573: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.

W1009 23:43:07.285009 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:573: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.

WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.
W1009 23:43:07.285009 15132 model_lib.py:574] Forced number of epochs for all eval validations to be 1.
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\utils\config_util.py:480: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

W1009 23:43:07.285009 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\utils\config_util.py:480: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:Maybe overwriting train_steps: None
I1009 23:43:07.285009 15132 config_util.py:480] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1
I1009 23:43:07.285009 15132 config_util.py:480] Maybe overwriting sample_1_of_n_eval_examples: 1
INFO:tensorflow:Maybe overwriting eval_num_epochs: 1
I1009 23:43:07.300634 15132 config_util.py:480] Maybe overwriting eval_num_epochs: 1
INFO:tensorflow:Maybe overwriting load_pretrained: True
I1009 23:43:07.300634 15132 config_util.py:480] Maybe overwriting load_pretrained: True
INFO:tensorflow:Ignoring config override key: load_pretrained
I1009 23:43:07.300634 15132 config_util.py:490] Ignoring config override key: load_pretrained
WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
W1009 23:43:07.316247 15132 model_lib.py:590] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu False
I1009 23:43:07.316247 15132 model_lib.py:623] create_estimator_and_inputs: use_tpu False, export_to_tpu False
INFO:tensorflow:Using config: {'_model_dir': 'training/trial_1/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x00000260DABBD288>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I1009 23:43:07.331873 15132 estimator.py:212] Using config: {'_model_dir': 'training/trial_1/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x00000260DABBD288>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x00000260DABB5708>) includes params argument, but params are not passed to Estimator.
W1009 23:43:07.331873 15132 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x00000260DABB5708>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Not using Distribute Coordinator.
I1009 23:43:07.331873 15132 estimator_training.py:186] Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
I1009 23:43:07.331873 15132 training.py:612] Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
I1009 23:43:07.347513 15132 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W1009 23:43:07.363146 15132 deprecation.py:323] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\data_decoders\tf_example_decoder.py:167: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W1009 23:43:07.363146 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\data_decoders\tf_example_decoder.py:167: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

W1009 23:43:07.363146 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\builders\dataset_builder.py:61: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

W1009 23:43:07.378762 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\builders\dataset_builder.py:61: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W1009 23:43:07.378762 15132 dataset_builder.py:66] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\builders\dataset_builder.py:80: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.parallel_interleave(...)`.
W1009 23:43:07.394386 15132 deprecation.py:323] From F:\Hujber\TensorFlow\models\research\object_detection\builders\dataset_builder.py:80: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.experimental.parallel_interleave(...)`.
WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\contrib\data\python\ops\interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
W1009 23:43:07.394386 15132 deprecation.py:323] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\contrib\data\python\ops\interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
2019-10-09 23:43:07.875217: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2019-10-09 23:43:08.008609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.815
pciBusID: 0000:41:00.0
2019-10-09 23:43:08.015807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2019-10-09 23:43:08.028156: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2019-10-09 23:43:08.034195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll
2019-10-09 23:43:08.040674: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll
2019-10-09 23:43:08.047902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll
2019-10-09 23:43:08.057279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll
2019-10-09 23:43:08.069618: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2019-10-09 23:43:08.072360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\anchor_generators\grid_anchor_generator.py:59: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W1009 23:43:13.023561 15132 deprecation.py:323] From F:\Hujber\TensorFlow\models\research\object_detection\anchor_generators\grid_anchor_generator.py:59: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.is_nan is deprecated. Please use tf.math.is_nan instead.

W1009 23:43:16.071676 15132 module_wrapper.py:139] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.is_nan is deprecated. Please use tf.math.is_nan instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\utils\ops.py:465: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W1009 23:43:16.149807 15132 deprecation.py:323] From F:\Hujber\TensorFlow\models\research\object_detection\utils\ops.py:465: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\utils\ops.py:468: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1009 23:43:16.149807 15132 deprecation.py:323] From F:\Hujber\TensorFlow\models\research\object_detection\utils\ops.py:468: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

W1009 23:43:17.891674 15132 module_wrapper.py:139] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

W1009 23:43:19.450865 15132 module_wrapper.py:139] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

W1009 23:43:19.466491 15132 module_wrapper.py:139] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.string_to_hash_bucket_fast is deprecated. Please use tf.strings.to_hash_bucket_fast instead.

W1009 23:43:21.049735 15132 module_wrapper.py:139] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.string_to_hash_bucket_fast is deprecated. Please use tf.strings.to_hash_bucket_fast instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\builders\dataset_builder.py:148: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.batch(..., drop_remainder=True)`.
W1009 23:43:21.471656 15132 deprecation.py:323] From F:\Hujber\TensorFlow\models\research\object_detection\builders\dataset_builder.py:148: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.batch(..., drop_remainder=True)`.
INFO:tensorflow:Calling model_fn.
I1009 23:43:21.487268 15132 estimator.py:1148] Calling model_fn.
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:162: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W1009 23:43:21.502909 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:162: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

2019-10-09 23:43:21.512084: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-10-09 23:43:21.529066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.815
pciBusID: 0000:41:00.0
2019-10-09 23:43:21.540071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2019-10-09 23:43:21.544621: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2019-10-09 23:43:21.547872: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll
2019-10-09 23:43:21.553238: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll
2019-10-09 23:43:21.557124: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll
2019-10-09 23:43:21.565002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll
2019-10-09 23:43:21.568175: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2019-10-09 23:43:21.572306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-10-09 23:43:22.169167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-09 23:43:22.172834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2019-10-09 23:43:22.176589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
2019-10-09 23:43:22.182536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 6269 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:41:00.0, compute capability: 7.5)
INFO:tensorflow:A GPU is available on the machine, consider using NCHW data format for increased speed on GPU.
I1009 23:43:22.178170 15132 nasnet.py:408] A GPU is available on the machine, consider using NCHW data format for increased speed on GPU.
WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
W1009 23:43:22.178170 15132 deprecation.py:323] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\slim\nets\nasnet\nasnet_utils.py:459: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W1009 23:43:22.287549 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\slim\nets\nasnet\nasnet_utils.py:459: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\core\anchor_generator.py:149: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.

W1009 23:43:29.248443 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\core\anchor_generator.py:149: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.

INFO:tensorflow:Scale of 0 disables regularizer.
I1009 23:43:29.248443 15132 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:986: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

W1009 23:43:29.248443 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:986: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

INFO:tensorflow:Scale of 0 disables regularizer.
I1009 23:43:29.264083 15132 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I1009 23:43:29.264083 15132 convolutional_box_predictor.py:148] depth of additional conv before box predictor: 0
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\box_coders\faster_rcnn_box_coder.py:82: The name tf.log is deprecated. Please use tf.math.log instead.

W1009 23:43:29.560967 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\box_coders\faster_rcnn_box_coder.py:82: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\core\minibatch_sampler.py:81: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.

W1009 23:43:29.592233 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\core\minibatch_sampler.py:81: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\utils\ops.py:1085: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
W1009 23:43:29.685991 15132 deprecation.py:506] From F:\Hujber\TensorFlow\models\research\object_detection\utils\ops.py:1085: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:185: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

W1009 23:43:29.701617 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:185: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
W1009 23:43:32.791057 15132 deprecation.py:323] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
INFO:tensorflow:Scale of 0 disables regularizer.
I1009 23:43:32.806683 15132 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I1009 23:43:32.822310 15132 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2235: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

W1009 23:43:32.837936 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2235: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2236: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
W1009 23:43:32.837936 15132 deprecation.py:323] From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2236: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\utils\variables_helper.py:126: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.

W1009 23:43:32.853562 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\utils\variables_helper.py:126: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:317: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

W1009 23:43:32.869188 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:317: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\core\losses.py:174: The name tf.losses.huber_loss is deprecated. Please use tf.compat.v1.losses.huber_loss instead.

W1009 23:43:35.818088 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\core\losses.py:174: The name tf.losses.huber_loss is deprecated. Please use tf.compat.v1.losses.huber_loss instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\core\losses.py:180: The name tf.losses.Reduction is deprecated. Please use tf.compat.v1.losses.Reduction instead.

W1009 23:43:35.818088 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\core\losses.py:180: The name tf.losses.Reduction is deprecated. Please use tf.compat.v1.losses.Reduction instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\core\losses.py:345: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See `tf.nn.softmax_cross_entropy_with_logits_v2`.

W1009 23:43:35.849340 15132 deprecation.py:323] From F:\Hujber\TensorFlow\models\research\object_detection\core\losses.py:345: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See `tf.nn.softmax_cross_entropy_with_logits_v2`.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2202: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

W1009 23:43:35.989976 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2202: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\builders\optimizer_builder.py:52: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.

W1009 23:43:36.021228 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\builders\optimizer_builder.py:52: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:359: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

W1009 23:43:36.021228 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:359: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:369: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W1009 23:43:36.021228 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:369: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:472: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W1009 23:43:48.484605 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:472: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:476: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.

W1009 23:43:49.816037 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:476: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.

WARNING:tensorflow:From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:477: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

W1009 23:43:49.816037 15132 module_wrapper.py:139] From F:\Hujber\TensorFlow\models\research\object_detection\model_lib.py:477: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

INFO:tensorflow:Done calling model_fn.
I1009 23:43:49.831650 15132 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I1009 23:43:49.831650 15132 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
I1009 23:43:59.346498 15132 monitored_session.py:240] Graph was finalized.
2019-10-09 23:43:59.354769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.815
pciBusID: 0000:41:00.0
2019-10-09 23:43:59.368323: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2019-10-09 23:43:59.372370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2019-10-09 23:43:59.376063: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll
2019-10-09 23:43:59.381228: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll
2019-10-09 23:43:59.384374: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll
2019-10-09 23:43:59.389889: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll
2019-10-09 23:43:59.393346: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2019-10-09 23:43:59.399778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-10-09 23:43:59.404838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-09 23:43:59.408820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2019-10-09 23:43:59.413834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
2019-10-09 23:43:59.418801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6269 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:41:00.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/trial_1/model.ckpt-0
I1009 23:43:59.424629 15132 saver.py:1284] Restoring parameters from training/trial_1/model.ckpt-0
WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\training\saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
W1009 23:44:04.970287 15132 deprecation.py:323] From C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\training\saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
INFO:tensorflow:Running local_init_op.
I1009 23:44:07.828599 15132 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I1009 23:44:09.031818 15132 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into training/trial_1/model.ckpt.
I1009 23:44:35.653282 15132 basic_session_run_hooks.py:606] Saving checkpoints for 0 into training/trial_1/model.ckpt.
2019-10-09 23:45:11.780278: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2019-10-09 23:45:14.790199: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2019-10-09 23:45:16.309576: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows
Relying on driver to perform ptx compilation. This message will be only logged once.

makecent commented 4 years ago

Same problem here. Ubuntu 20.04, TF 2.2.0, CUDA 10.1, cuDNN 7.6.5, GPU 1080

Yaffa16 commented 4 years ago

Same problem here with WIndows 10, TF 1.15 , CUDA 10.0.0 ,cudnn 7.6.5 , nvidia driver version : 416.16 , GPU 1070.

carykh commented 4 years ago

I'm having this exact same problem. I'm using TensorFlow-GPU 2.20, Windows 10, CUDA 10.1, cudnn 7.6. I read somewhere that this could be fixed by putting a symbolic link to wherever ptxas really is, but I checked with where ptxas, and it's the exact same folder as CUDA, so I am not sure what to do

mm0708 commented 4 years ago

Have this same problem on Windows 10, cudnn 7.6.5, cuda 10.1, tf-gpu 2.1.

Tensorflow seems to still run and I only get the warning once but the message still does appear on the first run of my script every time. Downgrading is unfortunately not an option so it would be nice if this error were fixed.

JoaoVictorDaijo commented 4 years ago

Have the same issue. The warning hangs for quite a few seconds then the program executes. It's using the GPU 'normally'. Not sure about performance since I never used Tensorflow before.

Windows 10 Version 10.0.19041 Build 19041 GTX 1060 on driver 451.48 cudnn-10.1-windows10-x64-v7.6.4.38 CUDA 10.1 Tensorflow-gpu 2.2.0

Not sure if relevant but I'm using a legacy code so I'm importing tensorflow like this: import tensorflow.compat.v1 as tf tf.disable_v2_behavior()

m-momeni commented 4 years ago

Having the same problem. python 3.8.3 TF 2.2.0 Windows10 Quadro T2000

abhyudayasrinet commented 4 years ago

I get this warning message but the training still continues.

2020-07-06 17:56:01.153426: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.

Python 3.8.3 TF 2.2.0 Windows 10 GeForce RTX 2070 Super

sjmikler commented 4 years ago

I have the same warning message and training continues.

Windows 10 Python 3.7.7 tensorflow 2.2.0 cudatoolkit 10.1 cudnn 7.6.5 nvidia driver 451.48 RTX 2080 Super max-Q

muhat commented 4 years ago

Windows 10 Enterprise x64 RAM 64.0GB CPU Intel(R) Xeon(R) E-2176G CPU @ 3.7GHz 3.70GHz GPU NVIDA GeForce RTX 2080 Ti CUDA 10.1 CuDNN 7.6.5.32 Tensorflow-gpu 1.15 Python 3.6.10

I got this message, and then it halted. W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows Relying on driver to perform ptx compilation. This message will be only logged once.

I solved this problem by adding the codes below.

from tensorflow import ConfigProto from tensorflow import InteractiveSession InteractiveSession(config = ConfigProto()

and it worked. The message still popped up, though. Hope it will be helpful.

ivankrylatskoe commented 4 years ago

Same issue. Windows 10 Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32 Tensorflow 2.2.0 CUDA 10.1 cudnn-10.1-windows10-x64-v7.6.5.32 GPU NVIDIA GeForce GTX 1060 6GB

AndroidDevelopersTools commented 4 years ago

any other solutions?

elazarg commented 4 years ago

Same issue. Windows 10 Tensorflow 2.3.0

murdockhou commented 4 years ago

windows 10 , tensorflow-gpu 2.2.0, occur:

2020-08-06 19:19:41.847317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.71GHz coreCount: 14 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 119.24GiB/s
2020-08-06 19:19:41.851554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-08-06 19:19:41.853742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-08-06 19:19:41.855913: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-08-06 19:19:41.858445: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-08-06 19:19:41.862867: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-08-06 19:19:41.864985: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-08-06 19:19:41.867453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-08-06 19:19:41.869781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-06 19:19:42.491446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-06 19:19:42.494204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0
2020-08-06 19:19:42.495520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N
2020-08-06 19:19:42.497226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2917 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-08-06 19:19:42.503450: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1e7eb3aeb90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-06 19:19:42.505993: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1650, Compute Capability 7.5
0it [00:00, ?it/s]2020-08-06 19:19:49.189187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-08-06 19:19:49.508514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-08-06 19:19:50.890988: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only

but no gpu used actually, everything seems run on CPU.

rafaelfaustini commented 4 years ago

Having same issues with Tensor Flow Version: 2.3.0; Keras Version: 2.4.0; Cuda 10, Cudnn 10; It identifies my gpu, when running a model it seems to be using gpu memory but my CUDAS get like 3%, doesn't seems to be a cpu bottleneck since it also reaches only a maximum of 15% when training the model

guomingjin commented 4 years ago

I faced the same issue. I used to train a model with Tensorflow 2.2.0 and 2.3.0 by using GPU. It worked fine a few days ago. But I just realized the inference speed is dramatically lower than ever. What's wrong with it? I would be grateful if anyone can help me out soon.

nioj commented 4 years ago

I am also getting the same warning, using Tensorflow 2.2 / 2.3 and Windows.

jpyrett commented 4 years ago

me too. We need to get this fixed. I'm not reformatting my laptop to use native GPU. GPU is working for other TF2. Just not when using ImageGenerator. It runs in CPU mode when training which is yuck since I have a 2080 Super card..

Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only

Poemtech commented 4 years ago

same issue. Tensorflow 2.2 Windows 10 Cuda 10.1 cudnn 8.0.2.39

FilloFolla commented 4 years ago

Same problem. Win 10 TensorFlow GPU 2.3.0 Cuda 10.0 (it's the same with 10.1) Cudnn 7.6.5.32 NVIDIA GeForce GTX 1050

Memory of GPU is almost all used, but GPU is 0% in use :-(

rybread1 commented 4 years ago

Same problem. Windows 10 TensorFlow GPU 2.3.0 Cuda 11.0 Cudnn 8.0.3.33 NVIDIA GeForce GTX 1070

Memory is almost 100% full, GPU is 0%. Model goes on to train using CPU.

bnavard commented 4 years ago

Is there any actual solution to this problem? I am having the same problem:

W: tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.

To give the solution to this issue the benefit of the doubt, I think the source of this problem is that TF ignores cuda bin directory defined in environment variable path wherein the ptxas file is based. And because of that, ptxas cannot be loaded into the program. However, there is a workaround by defining a symbolic link for the working directory representing the cuda bin directory.

The mentioned solution was for Unix based machines, but I am using Windows and have defined C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin in my environment variables path section and I think it works fine. Anyway, I still have the problem and don't know how to fix it.

RomanoBenedetto commented 4 years ago

Same problem. The problem is only in inference of the NN, during Training it use correctly the GPU. But this is A BIG PROBLEM, because inference go only using CPU. Please FIX THIS thanks a lot.

moh-na commented 4 years ago

Same Issue here.. Cant train any CNN without this appearing. Can we at least get a reason why this is happening?

yzy1996 commented 4 years ago

Same problem

RomanoBenedetto commented 4 years ago

Seems that the warning "Invoking ptxas not supported on Windows" do not preclude the use of the GPU. Actually so all working on my end with Windows 10, VS 2019, Tensorflow 2.3.0 cuda 11 cudnn 8 GPU is correctly used for training and in inference.

AMArostegui commented 4 years ago

I've checked and I can actually train a model using the GPU, after this message is thrown. Doesn't seem to be a problem.

All dll's are found, correctly loaded and CUDA is shown at 90% in the task manager, when training.

Windows 10, Python 3.6, CUDA 10.1, Tensorflow 2.3.0

Don't know about the inference, though.

hassannagy commented 4 years ago

for me i solve this problem pip install keras-gpu then tf-nightly pip install tf-nightly using CUDA 11.0 instead of 10.1 with CuDNN 8.0 for CUDA 11.0

platform specs

Windows 10 Tensorflow 2.3.1 Keras 2.4.3 Conda 4.9.0

aoberai commented 4 years ago

I have the same problem, when is it going to be fixed?

premthomas commented 4 years ago

Seems that the warning "Invoking ptxas not supported on Windows" do not preclude the use of the GPU. Actually so all working on my end with Windows 10, VS 2019, Tensorflow 2.3.0 cuda 11 cudnn 8 GPU is correctly used for training and in inference.

I believe that this is true. The message is a warning and not an error.

For testing, I disabled the GPU for my deep learning project using the code os.environ["CUDA_VISIBLE_DEVICES"] = "-1" and then ran training.

It took a significantly large amount of time than without explicitly disabling the GPU. Can someone confirm this?

roymiles commented 4 years ago

I had tf.config.optimizer.set_jit(True) in my code which worked on Linux but caused an error on Windows (the same error as described here). I found removing it resolved the issue (on Tensorflow v2.3).

dario-passos commented 4 years ago

Hi! I'm also having the same trouble on 2 different Windows 10 machines.

OS: Windows 10 19042.630 Python:3.6 Tensorflow: 2.3.0 CUDA:10.1 CuDNN:7.6

During compilation time of the tensorflow model I get:

2020-12-04 10:42:11.262923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-12-04 10:42:11.770550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-04 10:42:11.770733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2020-12-04 10:42:11.771638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-12-04 10:42:11.772184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6696 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-12-04 10:42:11.775255: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1cd151dd0d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-12-04 10:42:11.775344: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1070 Ti, Compute Capability 6.1
2020-12-04 10:42:16.934070: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-12-04 10:42:17.156466: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-12-04 10:42:17.749487: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.

So its detecting the GPU, loading the usual cuda dlls but then throwing the warning. In my case, the model still runs but on the CPU. In windows task manager GPU stays at 0.2% usage and no CUDA pane appears on the menu where we can see the GPU tasks (copy, 3D, etc...). This is actually what called my attention...

Doing echo %PATH% on the cmd shows that the CUDA directory where ptxas.exe resides is well defined. Help? Any progress on this issue?

@hassannagy you mentioned that you are using Cuda 11 and CuDNN 8? But on the Tensorflow tested configurations, Tensorflow tested configurations, that setup is not present. Does it works fine?

aulosea commented 3 years ago

When is this problem going to be fixed?

zhanggyarcher commented 3 years ago

Windows 10 1909 Python 3.7.9 tf-nightly 2.5.0 CUDA 11.0.2 cuDNN 8.05 RTX 3090

err

priyarana commented 3 years ago

same issue.

Windows 10 python 3.7.0 cuda 11.0 cuDNN 8.05 RTX 3070.

Waiting for the help

zhanggyarcher commented 3 years ago

I have fixed One solution is to download ptxas.exe to substitude C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\ptxas.exe. link：https://pan.baidu.com/s/1lirelifW3W9qIYvPImtCJg code：1234 link2: https://drive.google.com/file/d/1_RysxZ78HamK7kaTjbOlUaHJTvZZ2Xgn/view?usp=sharing

Another way is to try to downgrade pyzmq by pip install pyzmq==19.0.2, this may solve this issue

My environment: Windows 10 20H2 python 3.6.8 tensorflow 2.4.1 torch 1.7.0 cuda 11.0 cudnn 8.05 RTX 3090

@priyarana I would suggest you to pip install pyzmq==19.0.2, and substitude ptxas, these may solve the issue. I just reinstalled Windows 10 completely, and install python, cuda, cudnn, jupyterlab, tensorflow, and downgrade pyzmq to 19.0.2 and ptxas, and that problem disappeared.

priyarana commented 3 years ago

thank you so much @zhanggyarcher. I'll give that a try

swight-prc commented 3 years ago

Has anyone noticed that this warning only comes up with specific layer types? For example, I only get this warning on models involving LSTM layers.

DevLob-zz commented 3 years ago

what the Numpy Version u installed

priyarana commented 3 years ago

https://dobromyslova.medium.com/making-work-tensorflow-with-nvidia-rtx-3090-on-windows-10-7a38e8e582bf

This solved the problem completely. Hope that helps.

tensorflow / models