Closed disembarrasing closed 3 years ago
Hi @disembarrasing, It doesn't look like you've set TF_DETERMINISTIC_OPS='true'. Are you attempting to get GPU-deterministic operation? What's your reasoning behind submitting this issue to this repro/project?
Hi, I'm confused about this error, because I haven't changed anything inside the container, but it works differently on two separate systems. Moreover, as stated in the documentation, NGC TensorFlow Docker images, starting with version 19.06, implement GPU-deterministic op functionality. Version 19.12 (and beyond) also implements multi-algorithm deterministic cuDNN convolutions, which solves the problem of some layer configurations causing an exception to be thrown with the message "No algorithm worked!".
Is the issue I'm experiencing unrelated and I should adress this issue somewhere else?
It's cool that you found your way to this repo. That documentation did come from me. However, it's supposed to be referring to the functionality that is enabled when TF_DETERMINISTIC_OPS
is set to true
or 1
(perhaps that's ambiguous). When not enabling op determinism, there are other potential causes of that exception being thrown. It means that TensorFlow was unable (for some reason) to find an appropriate cuDNN convolution algorithm (for functional_1/resnet50/conv1_conv/Conv2D
in the specific example that you've provided).
I could dig into this, but it's almost certainly not related to framework determinism in TensorFlow. I'm figuring out where a more appropriate place to report this issue might be.
Meanwhile, I have a question about your setup. I see that the 2070, the hardware on which you're seeing this exception, has 8GB of memory. How much memory is on the 1060, the hardware on which you don't get the exception?
Thank you for the reply. The 1060 I'm using has 6GB of memory.
Wonderful. That's a useful datapoint. Would you be able to run this script for me inside a stock TensorFlow container to see if this issue repros there? (I don't think it references anything NGC-specific.) If so, try using tensorflow/tensorflow:2.3.0-gpu
.
The issue does not occur, here's full output:
2020-10-02 23:24:49.057271: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Found 3745 files belonging to 2 classes.
2020-10-02 23:24:49.839316: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-10-02 23:24:49.878143: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 23:24:49.878517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:26:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5
coreClock: 1.77GHz coreCount: 40 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-10-02 23:24:49.878532: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-02 23:24:49.879474: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-02 23:24:49.880414: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-02 23:24:49.880558: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-02 23:24:49.881470: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-02 23:24:49.881986: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-02 23:24:49.883908: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-02 23:24:49.883997: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 23:24:49.884399: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 23:24:49.884723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-02 23:24:49.884953: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-02 23:24:49.889046: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3600015000 Hz
2020-10-02 23:24:49.889588: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x552c280 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-02 23:24:49.889609: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-10-02 23:24:49.975379: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 23:24:49.975793: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x552e680 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-10-02 23:24:49.975815: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2070 SUPER, Compute Capability 7.5
2020-10-02 23:24:49.976053: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 23:24:49.976628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:26:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5
coreClock: 1.77GHz coreCount: 40 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-10-02 23:24:49.976658: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-02 23:24:49.976678: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-02 23:24:49.976695: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-02 23:24:49.976715: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-02 23:24:49.976731: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-02 23:24:49.976745: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-02 23:24:49.976766: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-02 23:24:49.976861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 23:24:49.977468: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 23:24:49.977996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-02 23:24:49.978027: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-02 23:24:50.253653: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-02 23:24:50.253683: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-10-02 23:24:50.253687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-10-02 23:24:50.253848: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 23:24:50.254221: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 23:24:50.254545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7020 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 SUPER, pci bus id: 0000:26:00.0, compute capability: 7.5)
Found 2402 files belonging to 2 classes.
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5
94773248/94765736 [==============================] - 3s 0us/step
2020-10-02 23:24:54.890545: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2020-10-02 23:24:54.890581: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1391] Profiler found 1 GPUs
2020-10-02 23:24:54.891008: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcupti.so.10.1
2020-10-02 23:24:54.991597: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1441] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES
2020-10-02 23:24:55.733489: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-02 23:24:56.972247: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
32/61 [==============>...............] - ETA: 2s - loss: 0.7563 - accuracy: 0.4346
Another awesome datapoint. Thanks. tensorflow/tensorflow:2.3.0-gpu
uses CUDA 10.1 / cuDNN 7.6 whilst the NGC container uses CUDA 11.0 / cuDNN 8.0.4. This could very well be related to a bug in one of those products. I'll get back to you when I know how to proceed with getting this to the appropriate people.
I've filed an internal bug report for this problem and I've linked it back to this GitHub issue.
I can reproduce using the 20.09-tf2 NGC container. A workaround is to enable allow_growth by exporting the following.
TF_FORCE_GPU_ALLOW_GROWTH=true
Using a development build of the upcoming 20.10-tf2 NGC container, I can no longer reproduce the failure (regardless of allow_growth setting).
@disembarrasing, please will you confirm that @nluehr's workaround (described above) solves your problem?
I've tested the workaround and it works, thank you both! Would you mind if I close the issue after testing 20.10-tf2? I'd like to be 100% sure the bug is gone.
Wonderful. Yes, it's okay to leave this issue open until you've confirmed that the problem is also solved in 20.10-tf2
(without using the workaround).
I've tested the latest version of ngc container and the problem is solved, thank you @nluehr and @duncanriach.
Prerequisites
nvcr.io/nvidia/tensorflow:20.09-tf2-py3
withnvidia-container-toolkit
installed on hostUsed script
Run command
docker run --gpus all --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 --rm -it -v /home/kw/DATASETS/liveness:/home/kw/DATASETS/liveness -v /home/kw/PycharmProjects/liveness_resnet/:/home/kw/PycharmProjects/liveness_resnet --name tf_2009 nvcr.io/nvidia/tensorflow:20.09-tf2-py3
Bug description
After running above script, an exception with "No algorithm worked!" message is thrown.
Full output
System information
Additional information
Script provided above works perfectly on a twin configuration (same OS, Docker image, Docker version and OS driver version), but with GTX 1060.