Open j-paulus opened 3 years ago
Update: The exact same problem is present in ONNX Runtime 1.6.0.
Update: The problem persists in ONNX Runtime 1.7.0 with DML 1.4.2.
@j-paulus Sorry this got buried in the notification noise - I'm cleaning up old issues. :/ I was able to repro seeing a C++ Ort::Exception::Exception
with ORT 1.10 via the value 768 on Nvidia Quadro P400 (thanks for the repro snippets). The output_tensors
is size 1, but the contained Ort::Value
is empty. After enabling the debug layer via dxcpl.exe...
...I saw the following output:
D3D12 ERROR: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason
DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung.
As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered.
The current Device Context was executing commands when the hang occurred.
The application may want to respawn and fallback to less aggressive use of the display hardware).
[ EXECUTION ERROR #232: DEVICE_REMOVAL_PROCESS_AT_FAULT]
I suspect the ConvTranspose is taking too long.
Describe the bug I am trying to use DirectML EP for accelerating the network computations. The system loads the model and runs it multiple times, each time with a different input. When running on CPU, this works fine. When running on CUDA EP under Linux, this works fine. When running this on CUDA EP under Windows, there is a significant delay in the start of the processing, but eventually it runs. Using DML EP, the execution crashes on the second call to Session::Run().
This crash seems to depend from the model size. Using the example model below, the crash happens with kernel size of 512x512, but not with 384x384 (these numbers are probably valid only on my system). So a bigger model crashes. This would somehow suggest some memory handling issue.
I am using the dll of 1.5.2 available under releases.
Urgency
System information
To Reproduce
Create model with pytorch (1.7):
C-program to run the model (linked against dll:
Expected behavior Expecting multiple calls to Session::Run() to work without crashing also with DML.
Screenshots If applicable, add screenshots to help explain your problem.
Additional context Add any other context about the problem here. If the issue is about a particular model, please share the model details as well to facilitate debugging.