makecent
commented
4 years ago Same problem here. Ubuntu 20.04, TF 2.2.0, CUDA 10.1, cuDNN 7.6.5, GPU 1080
Open edwardHujber opened 4 years ago
makecent
commented
4 years ago Same problem here. Ubuntu 20.04, TF 2.2.0, CUDA 10.1, cuDNN 7.6.5, GPU 1080
Yaffa16
commented
4 years ago Same problem here with WIndows 10, TF 1.15 , CUDA 10.0.0 ,cudnn 7.6.5 , nvidia driver version : 416.16 , GPU 1070.
carykh
commented
4 years ago I'm having this exact same problem. I'm using TensorFlow-GPU 2.20, Windows 10, CUDA 10.1, cudnn 7.6. I read somewhere that this could be fixed by putting a symbolic link to wherever ptxas really is, but I checked with where ptxas, and it's the exact same folder as CUDA, so I am not sure what to do
mm0708
commented
4 years ago Have this same problem on Windows 10, cudnn 7.6.5, cuda 10.1, tf-gpu 2.1.
Tensorflow seems to still run and I only get the warning once but the message still does appear on the first run of my script every time. Downgrading is unfortunately not an option so it would be nice if this error were fixed.
JoaoVictorDaijo
commented
4 years ago Have the same issue. The warning hangs for quite a few seconds then the program executes. It's using the GPU 'normally'. Not sure about performance since I never used Tensorflow before.
Windows 10 Version 10.0.19041 Build 19041 GTX 1060 on driver 451.48 cudnn-10.1-windows10-x64-v7.6.4.38 CUDA 10.1 Tensorflow-gpu 2.2.0
Not sure if relevant but I'm using a legacy code so I'm importing tensorflow like this: import tensorflow.compat.v1 as tf tf.disable_v2_behavior()
m-momeni
commented
4 years ago Having the same problem. python 3.8.3 TF 2.2.0 Windows10 Quadro T2000
abhyudayasrinet
commented
4 years ago I get this warning message but the training still continues.
2020-07-06 17:56:01.153426: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.Python 3.8.3 TF 2.2.0 Windows 10 GeForce RTX 2070 Super
sjmikler
commented
4 years ago I have the same warning message and training continues.
Windows 10 Python 3.7.7 tensorflow 2.2.0 cudatoolkit 10.1 cudnn 7.6.5 nvidia driver 451.48 RTX 2080 Super max-Q
muhat
commented
4 years ago Windows 10 Enterprise x64 RAM 64.0GB CPU Intel(R) Xeon(R) E-2176G CPU @ 3.7GHz 3.70GHz GPU NVIDA GeForce RTX 2080 Ti CUDA 10.1 CuDNN 7.6.5.32 Tensorflow-gpu 1.15 Python 3.6.10
I got this message, and then it halted.
W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows Relying on driver to perform ptx compilation. This message will be only logged once.
I solved this problem by adding the codes below.
from tensorflow import ConfigProto
from tensorflow import InteractiveSession
InteractiveSession(config = ConfigProto()
and it worked. The message still popped up, though. Hope it will be helpful.
ivankrylatskoe
commented
4 years ago Same issue. Windows 10 Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32 Tensorflow 2.2.0 CUDA 10.1 cudnn-10.1-windows10-x64-v7.6.5.32 GPU NVIDIA GeForce GTX 1060 6GB
AndroidDevelopersTools
commented
4 years ago any other solutions?
elazarg
commented
4 years ago Same issue. Windows 10 Tensorflow 2.3.0
murdockhou
commented
4 years ago windows 10 , tensorflow-gpu 2.2.0, occur:
2020-08-06 19:19:41.847317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.71GHz coreCount: 14 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 119.24GiB/s
2020-08-06 19:19:41.851554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-08-06 19:19:41.853742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-08-06 19:19:41.855913: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-08-06 19:19:41.858445: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-08-06 19:19:41.862867: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-08-06 19:19:41.864985: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-08-06 19:19:41.867453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-08-06 19:19:41.869781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-06 19:19:42.491446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-06 19:19:42.494204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-08-06 19:19:42.495520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-08-06 19:19:42.497226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2917 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-08-06 19:19:42.503450: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1e7eb3aeb90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-06 19:19:42.505993: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1650, Compute Capability 7.5
0it [00:00, ?it/s]2020-08-06 19:19:49.189187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-08-06 19:19:49.508514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-08-06 19:19:50.890988: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms onlybut no gpu used actually, everything seems run on CPU.
rafaelfaustini
commented
4 years ago Having same issues with Tensor Flow Version: 2.3.0; Keras Version: 2.4.0; Cuda 10, Cudnn 10; It identifies my gpu, when running a model it seems to be using gpu memory but my CUDAS get like 3%, doesn't seems to be a cpu bottleneck since it also reaches only a maximum of 15% when training the model
guomingjin
commented
4 years ago I faced the same issue. I used to train a model with Tensorflow 2.2.0 and 2.3.0 by using GPU. It worked fine a few days ago. But I just realized the inference speed is dramatically lower than ever. What's wrong with it? I would be grateful if anyone can help me out soon.
nioj
commented
4 years ago I am also getting the same warning, using Tensorflow 2.2 / 2.3 and Windows.
jpyrett
commented
4 years ago me too. We need to get this fixed. I'm not reformatting my laptop to use native GPU. GPU is working for other TF2. Just not when using ImageGenerator. It runs in CPU mode when training which is yuck since I have a 2080 Super card..
Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Poemtech
commented
4 years ago same issue. Tensorflow 2.2 Windows 10 Cuda 10.1 cudnn 8.0.2.39
FilloFolla
commented
4 years ago Same problem. Win 10 TensorFlow GPU 2.3.0 Cuda 10.0 (it's the same with 10.1) Cudnn 7.6.5.32 NVIDIA GeForce GTX 1050
Memory of GPU is almost all used, but GPU is 0% in use :-(
rybread1
commented
4 years ago Same problem. Windows 10 TensorFlow GPU 2.3.0 Cuda 11.0 Cudnn 8.0.3.33 NVIDIA GeForce GTX 1070
Memory is almost 100% full, GPU is 0%. Model goes on to train using CPU.
bnavard
commented
4 years ago Is there any actual solution to this problem? I am having the same problem:
W: tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.To give the solution to this issue the benefit of the doubt, I think the source of this problem is that TF ignores cuda bin directory defined in environment variable path wherein the ptxas file is based. And because of that, ptxas cannot be loaded into the program. However, there is a workaround by defining a symbolic link for the working directory representing the cuda bin directory.
The mentioned solution was for Unix based machines, but I am using Windows and have defined C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin in my environment variables path section and I think it works fine. Anyway, I still have the problem and don't know how to fix it.
RomanoBenedetto
commented
4 years ago Same problem. The problem is only in inference of the NN, during Training it use correctly the GPU. But this is A BIG PROBLEM, because inference go only using CPU. Please FIX THIS thanks a lot.
moh-na
commented
4 years ago Same Issue here.. Cant train any CNN without this appearing. Can we at least get a reason why this is happening?
yzy1996
commented
4 years ago Same problem
RomanoBenedetto
commented
4 years ago Seems that the warning "Invoking ptxas not supported on Windows" do not preclude the use of the GPU. Actually so all working on my end with Windows 10, VS 2019, Tensorflow 2.3.0 cuda 11 cudnn 8 GPU is correctly used for training and in inference.
AMArostegui
commented
4 years ago I've checked and I can actually train a model using the GPU, after this message is thrown. Doesn't seem to be a problem.
All dll's are found, correctly loaded and CUDA is shown at 90% in the task manager, when training.
Windows 10, Python 3.6, CUDA 10.1, Tensorflow 2.3.0
Don't know about the inference, though.
hassannagy
commented
3 years ago for me i solve this problem
pip install keras-gpu
then tf-nightly
pip install tf-nightly
using CUDA 11.0 instead of 10.1
with CuDNN 8.0 for CUDA 11.0
Windows 10 Tensorflow 2.3.1 Keras 2.4.3 Conda 4.9.0
aoberai
commented
3 years ago I have the same problem, when is it going to be fixed?
premthomas
commented
3 years ago Seems that the warning "Invoking ptxas not supported on Windows" do not preclude the use of the GPU. Actually so all working on my end with Windows 10, VS 2019, Tensorflow 2.3.0 cuda 11 cudnn 8 GPU is correctly used for training and in inference.
I believe that this is true. The message is a warning and not an error.
For testing, I disabled the GPU for my deep learning project using the code
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
and then ran training.
It took a significantly large amount of time than without explicitly disabling the GPU. Can someone confirm this?
roymiles
commented
3 years ago I had tf.config.optimizer.set_jit(True) in my code which worked on Linux but caused an error on Windows (the same error as described here). I found removing it resolved the issue (on Tensorflow v2.3).
dario-passos
commented
3 years ago Hi! I'm also having the same trouble on 2 different Windows 10 machines.
OS: Windows 10 19042.630 Python:3.6 Tensorflow: 2.3.0 CUDA:10.1 CuDNN:7.6
During compilation time of the tensorflow model I get:
2020-12-04 10:42:11.262923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-12-04 10:42:11.770550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-04 10:42:11.770733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-12-04 10:42:11.771638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-12-04 10:42:11.772184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6696 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-12-04 10:42:11.775255: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1cd151dd0d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-12-04 10:42:11.775344: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1070 Ti, Compute Capability 6.1
2020-12-04 10:42:16.934070: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-12-04 10:42:17.156466: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-12-04 10:42:17.749487: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.So its detecting the GPU, loading the usual cuda dlls but then throwing the warning. In my case, the model still runs but on the CPU. In windows task manager GPU stays at 0.2% usage and no CUDA pane appears on the menu where we can see the GPU tasks (copy, 3D, etc...). This is actually what called my attention...
Doing echo %PATH% on the cmd shows that the CUDA directory where ptxas.exe resides is well defined.
Help? Any progress on this issue?
@hassannagy you mentioned that you are using Cuda 11 and CuDNN 8? But on the Tensorflow tested configurations, Tensorflow tested configurations, that setup is not present. Does it works fine?
aulosea
commented
3 years ago When is this problem going to be fixed?
zhanggyarcher
commented
3 years ago Windows 10 1909 Python 3.7.9 tf-nightly 2.5.0 CUDA 11.0.2 cuDNN 8.05 RTX 3090
priyarana
commented
3 years ago same issue.
Windows 10 python 3.7.0 cuda 11.0 cuDNN 8.05 RTX 3070.
Waiting for the help
zhanggyarcher
commented
3 years ago I have fixed One solution is to download ptxas.exe to substitude C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\ptxas.exe. link：https://pan.baidu.com/s/1lirelifW3W9qIYvPImtCJg code：1234 link2: https://drive.google.com/file/d/1_RysxZ78HamK7kaTjbOlUaHJTvZZ2Xgn/view?usp=sharing
Another way is to try to downgrade pyzmq by pip install pyzmq==19.0.2, this may solve this issue
My environment: Windows 10 20H2 python 3.6.8 tensorflow 2.4.1 torch 1.7.0 cuda 11.0 cudnn 8.05 RTX 3090
@priyarana I would suggest you to pip install pyzmq==19.0.2, and substitude ptxas, these may solve the issue. I just reinstalled Windows 10 completely, and install python, cuda, cudnn, jupyterlab, tensorflow, and downgrade pyzmq to 19.0.2 and ptxas, and that problem disappeared.
priyarana
commented
3 years ago thank you so much @zhanggyarcher. I'll give that a try
swight-prc
commented
3 years ago Has anyone noticed that this warning only comes up with specific layer types? For example, I only get this warning on models involving LSTM layers.
DevLob-zz
commented
3 years ago what the Numpy Version u installed
priyarana
commented
3 years ago This solved the problem completely. Hope that helps.
System information
What is the top-level directory of the model you are using:
\models\research\object_detection\
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): NO, trying to use model_main.py
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): v1.15.0-rc2-10-g38ea9bbfea 1.15.0-rc3
Bazel version (if compiling from source): N/A
CUDA/cuDNN version: CUDA Version 10.0.130 cuDNN: 7.6.4.38
GPU model and memory: GeForce RTX 2080 SUPER. 8 GB dedicated, 32 GB shared
Exact command to reproduce: From within an Anaconda environment:
python model_main.py --alsologtostderr --model_dir=training/trial_1/ --pipeline_config_path=training/trial_1/faster_rcnn_nas_coco.configDescribe the problem
Hangs on a
message. Sits there forever. Sometimes (usually after restarting the terminal and clearing out any produced files like .ckpt and .pbtxt ) it gets passed this point and soon after crashes with an out of memory problem. Mentioning that because I don't know if its related or not.
Source code / logs