Unable to train on a single GPU - ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError

[ikram-md]

Hello, i am trying to train the model on Visual Genome on a single GPU. About the GPU :

• GeForce RTX 3060, 12Go RAM
• CUDA Version: 12.0
• Attached GPU : 1 I am working in base environment with:
• torch : 2.3.0
• python : 3.11.7 And when i run nvcc --version i get the following :

  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2022 NVIDIA Corporation
  Built on Wed_Sep_21_10:33:58_PDT_2022
  Cuda compilation tools, release 11.8, V11.8.89
  Build cuda_11.8.r11.8/compiler.31833905_0

Since i only want to train on the current GPU i have, i am running the following command :

python -m torch.distributed.launch --nproc_per_node=1 --use_env main.py --dataset vg --img_folder data/vg/images/ --ann_path data/vg/ --batch_size 1 --output_dir ckpt

And i am getting the following error :

I tried reducing the batch size to 1 ( one image per GPU ) and get the number of process per node also to be 1 and i am still getting the same error, i tried reading about the error but it's difficult to find since it depends on the case.

[ikram-md]

Solved the problem by starting a new virtual environment, installing cuda & pytorch from the official documentation Start locally then installing python 3.9, cython and all the necessary dependencies. The caveat is that cuda version must be exactly compatible with your GPU.

[wuzhiwei2001]

Solved the problem by starting a new virtual environment, installing cuda & pytorch from the official documentation Start locally then installing python 3.9, cython and all the necessary dependencies. The caveat is that cuda version must be exactly compatible with your GPU.

Hello, I encountered the same issue as you did and followed your instructions for installation thoroughly. However, the problem still persists. Could you please let me know which versions of PyTorch and CUDA you selected? I would greatly appreciate your response!