Closed rohan-varma closed 4 months ago
@rohan-varma I spent some time trying to debug this last week as well and what I found was that irrespective of what CUDA_AVAILABLE_DEVICE
is set as, local_rank
is always 0 and so get_device
actually handles this the right way i.e. the run is kicked off on the right device. I do agree that this is very confusing though. Something that we need to look at while we revamp tune run
, @joecummings
Launching a ft with
And entering a
pdb
after call toget_model
and inspectingself._device
yields the following:This seems to be because
torchtune.utils.device.get_device
does not handle non-zero CUDA ordinals.