above is my code and
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.23.04 Driver Version: 455.23.04 CUDA Version: 11.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 3090 On | 00000000:09:00.0 On | N/A |
| 33% 53C P2 111W / 350W | 1016MiB / 24265MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
this is nvidia-smi
I use tensorflow-gpu 2.2 and cuda toolkit 10.1 and cudnn 7.6
My computer is 3900X 128GB(RAM) RTX3090 500GB(SSD)
and if run above code error message is below.
File "/home/sentiment/anaconda3/envs/mybert/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:GPU:0 to /job:localhost/replica:0/task:0/device:CPU:0 in order to run Identity: GPU sync failed [Op:Identity]
I want to train albert finetuning.
if i use tensorflow for cpu. it work fine but 1 epoch per 6 hour for training.
so I hope to use gpu
I really hard to find out solution for fixing but failed.
above is my code and +-----------------------------------------------------------------------------+ | NVIDIA-SMI 455.23.04 Driver Version: 455.23.04 CUDA Version: 11.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce RTX 3090 On | 00000000:09:00.0 On | N/A | | 33% 53C P2 111W / 350W | 1016MiB / 24265MiB | 1% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ this is nvidia-smi
I use tensorflow-gpu 2.2 and cuda toolkit 10.1 and cudnn 7.6 My computer is 3900X 128GB(RAM) RTX3090 500GB(SSD)
and if run above code error message is below.
File "/home/sentiment/anaconda3/envs/mybert/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:GPU:0 to /job:localhost/replica:0/task:0/device:CPU:0 in order to run Identity: GPU sync failed [Op:Identity]
I want to train albert finetuning. if i use tensorflow for cpu. it work fine but 1 epoch per 6 hour for training. so I hope to use gpu
I really hard to find out solution for fixing but failed.
is there anyone know how to fix this error?