2024-02-23 17:53:11.979467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2024-02-23 17:54:07.115039: W tensorflow/core/common_runtime/bfc_allocator.cc:309] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
box = tf.Tensor(
[[[nan nan nan nan]
[nan nan nan nan]
[nan nan nan nan]
...
For full information, please refer to the attachment.
Environment: Ubuntu 22.02, Python 3.7, tf-gpu==2.1.0
2024-02-23 17:53:11.979467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2024-02-23 17:54:07.115039: W tensorflow/core/common_runtime/bfc_allocator.cc:309] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
box = tf.Tensor(
[[[nan nan nan nan]
[nan nan nan nan]
[nan nan nan nan] ...
For full information, please refer to the attachment.
6359E72E-4CAD-4ce3-99EC-07FDFA630C4D.txt