Open CharlelieLrt opened 1 year ago
Demangled:
I don't know what the at
namespace is, but I assume it's part of PyTorch. Therefore this seems to be slightly different than #824 which reported duplicates in Thrust.
I'm not sure where to go for debugging this one, would appreciate a response from someone who knows the Realm CUDA module.
I believe the at
namespace is Pytorch's ATen tensor library.
You can try to either use the cpu-only pytorch or disable the cuda hijack by setting -DLegion_USE_CUDART_HIJACK=OFF
FYI, we may need to do something to make Regent work with the hijack disabled (assuming this is a combined Regent+Pygion code). I'll need @magnatelee to remind me of the details.
@eddy16112 thanks for the suggestions. I have been using CPU-only Pytorch in this context quite a lot and it is working just fine... but CPU-only is now insufficient for what I want to do.
@elliottslaughter yes it is a Regent + Pygion code.
@elliottslaughter we can use these two calls to get a cuda stream for launching tasks https://gitlab.com/StanfordLegion/legion/-/blob/master/test/realm/task_stream.cc#L51-52. It should avoid the context sync that are used when disabling cuda hijack.
Just a precision (if that matters): I do not execute any Pytorch kernel on GPUs shared with Legion. Pytorch kernels are only executed on dedicated GPUs (with the -ll:skigpus
option)
You can go ahead and try Regent without the hijack if you want. I don't remember exactly what the issue is, but I am guessing it will break, and for reasons unrelated to GPU usage / -ll:skipgpus
. But without trying it, we won't know.
Ok I'll try without the hijack. Is it just a runtime flag or do I need to re-build Legion with the hijack disabled?
You have to re-build Legion. If you are using cmake, you can disable hijack by setting -DLegion_HIJACK_CUDART=OFF
I made a slightly different reproducer to reflect the fact that the code giving these warnings is a Regent + Pygion code (and also to test the effect of disabling the Cuda hijack on the Regent part of the code).
main.rg
:
import 'regent'
local nb_gpus = 3
local format = require("std/format")
__demand(__leaf)
extern task ReductionPytorch(input: region(ispace(int1d), double)): double
where
reads(input)
end
ReductionPytorch:set_task_id(9000)
__demand(__cuda, __leaf)
task ReductionRegent(input: region(ispace(int1d), double),
output: region(ispace(int1d), double))
where
reads(input),
reads writes(output)
do
__demand(__openmp)
for c in input do
output[output.bounds.lo] += input[c]
end
end
__demand(__leaf)
task checkSum(output: region(ispace(int1d), double))
where
reads(output)
do
for c in output do
format.println("Regent sum at {} = {}", c, output[c])
end
end
__demand(__inner)
__forbid(__replicable)
task main()
var points_input_Rg = ispace(int1d, {32 * nb_gpus})
var points_output_Rg = ispace(int1d, {nb_gpus})
var tiles_Rg = ispace(int1d, {nb_gpus})
var input_Rg = region(points_input_Rg, double)
var output_Rg = region(points_output_Rg, double)
var tiles_input_Rg = partition(equal, input_Rg, tiles_Rg)
var tiles_output_Rg = partition(equal, output_Rg, tiles_Rg)
var points_input_Py = ispace(int1d, {32})
var input_Py = region(points_input_Py, double)
var output_Py: double
__demand(__index_launch)
for c in tiles_Rg do
fill((tiles_input_Rg[c]), 1.0)
end
__demand(__index_launch)
for c in tiles_Rg do
fill((tiles_output_Rg[c]), 0.0)
end
fill(input_Py, 1.0)
__demand(__index_launch)
for c in tiles_Rg do
ReductionRegent(tiles_input_Rg[c], tiles_output_Rg[c])
end
output_Py = ReductionPytorch(input_Py)
__demand(__index_launch)
for c in tiles_Rg do
checkSum(tiles_output_Rg[c])
end
end
regentlib.start(main)
python_main.py
:
from pygion import task, R, Region, float64
import torch
@task(task_id=9000,
argument_types=[Region],
privileges=[R],
return_type=float64,
leaf=True,
calling_convention='regent')
def ReductionPytorch(input_Py):
device = torch.device("cuda:0")
data = torch.from_numpy(input_Py.__value)
data = data.to(device)
s = torch.sum(data)
output_Py = s.item()
print(f"Pytorch sum = {output_Py}", flush=True)
return output_Py
I ran the reproducer on a single node of a machine with 4 GPUs/node with: regent.py main.rg -ll:py 1 -ll:pyimport python_main -ll:cpu 1 -ll:gpu 3 -cuda:skipgpus 1
1. With Cuda hijack enabled:
I am getting all the duplicate registration
warnings above. Then the Python part of the code fails with:
[0 - 20004276f8b0] 12.134474 {6}{cudart}: cudaGetDevice() called outside CUDA task
2. With Cuda hijack disabled: The warnings above don't occur and the code completes successfully.
Is the point 2 above expected? I thought Regent would not work without the Cuda hijack?
It's been a while since I looked so I suppose Regent works without the hijack now. Good to know.
I think there is still a potential performance issue around setting the correct stream, so let's keep this open for now. But it should (hopefully) be usable for you.
I am importing Pytorch with Cuda enabled into a Pygion script
hello.py
, and this gives me a lot of "duplicate registration of function" warning messages. Pytorch is version 1.12.0 built from source with Cuda 11.6.hello.py
:And this is ran with
legion_python -ll:py 1 -ll:gpu 1
This gives a lot of warnings:Obviously the warnings don't occur if I remove the
import torch
. The warnings do not seem to prevent the script from running though, so I am wondering if I can simply disregard those or if it is something that should be addressed.Edit: I realized this issue was already reported long time ago (#824) @eddy16112 @streichler I know that this issue is old, but could you identify the source of the problem at that time?