search

mileyan / pseudo_lidar

(CVPR 2019) Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving
https://mileyan.github.io/pseudo_lidar/
MIT License
975 stars 216 forks source link

There is an imbalance between your GPUs #8

Closed kotchin closed 5 years ago

kotchin commented 5 years ago

When executing the step "train psmnet with 4 TITAN X GPUs" there's an annoying warning caused by the necessity of using Python2.7 and pytorch, in full:

torch/nn/parallel/data_parallel.py:25: UserWarning: There is an imbalance between your GPUs. You may want to exclude GPU 1 which has less than 75% of the memory or cores of GPU 0. You can do so by setting the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES environment variable.

I suspect it could prevent the usage of multi-gpu. Luckily, there's an easy fix for it. Go to the file in question causing the warning (torch/nn/parallel/data_parallel.py), and at the top of the file, add the following line:

from __future__ import division

Try again. In case you run into the following issue:

ImportError: No module named future

Just install future, e.g. using one of the following commands:

conda install future
pip3 install future
mileyan commented 5 years ago

The warning shows your GPU 1 is different with your GPU 0. Could you please list your hardware environment?

kotchin commented 5 years ago

It says they are different yes, but they aren't. I'm running with 2x RTX 2080 Ti, with the same amount of memory.

I listed above the solution to fix it, which seems to be a problem with Python2 divisions.

I found this fix here.

EDIT: There is nothig wrong with your code, it's a pytorch error. I'm just sharing the fix which seems to have touched other people too.