and i am trying to train the bert model using this command:
! bash train.sh path_to_data model_path log_path -visible_gpus -1
and I have tried this also
! bash train.sh path_to_data model_path log_path
and I am getting the same problem:
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
I have downloaded a data from: https://drive.google.com/file/d/1DN7ClZCCXsk2KegmC6t4ClBwtAf5galI/view
and i am trying to train the bert model using this command: ! bash train.sh path_to_data model_path log_path -visible_gpus -1
and I have tried this also ! bash train.sh path_to_data model_path log_path
and I am getting the same problem: RuntimeError: CUDA error: invalid device ordinal CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.I am using gpu T4