[Tensorflow 2.8.1-gpu] Training not accelerated with NVIDIA GPU

[kmittman]

[moving comment to Tensorflow]

Hey @kmittman,

I hope you could help me with this as well. I'm training my Siamese Network for Face verification using TensorFlow, below are the time it is taking for training for comparison with CPU and GPU:

Siamese on CPU : 18/899 [..............................] - ETA: 4:41:43 - loss: 0.8343 Siamese on GPU: 18/899 [..............................] - ETA: 4:52:04 - loss: 0.9894

It seems both of them take exactly similar time, which shouldn't be the case. Below i'm also attaching CPU and GPU details:

[name: "/device:CPU:0" device_type: "CPU" memory_limit: 268435456 locality { } incarnation: 3939414009020902295 xla_global_id: -1 , name: "/device:GPU:0" device_type: "GPU" memory_limit: 14258995200 locality { bus_id: 1 links { } } incarnation: 3474452779198683813 physical_device_desc: "device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5" xla_global_id: 416903419 ]

These are following commands i'm running at the training time in order to make sure it is working on GPU:

from tensorflow.python.client import device_lib print(device_lib.list_local_devices()) print(tf.config.list_physical_devices('GPU')) tf.debugging.set_log_device_placement(True)

As disccused earlier i'm using Tensorflow-2.8.1-gpu for training it on GPU. Am i doing anything wrong here or do I need to run some other commands as well in order to fix this ?

Originally posted by @Nrohlable in https://github.com/NVIDIA/cuda-repo-management/issues/7#issuecomment-1252108439

[kmittman]

FYI @Nrohlable created a new issue for your comment.