Closed fxmarty closed 5 months ago
cc @tianleiwu @yufenglee
I guess one may compile with ORT_DEBUG_NODE_IO_DUMP_SHAPE_DATA=1
to see which node the issue comes from.
This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.
Hi @yufenglee @tianleiwu, this issue is not stale and reported by user for an other architecture (table-transformer), with onnxruntime-gpu==1.17.1
: https://github.com/huggingface/optimum/issues/1774
The issue is resolved in the main branch.
I did reproduce it in 1.17.1:
So the issue is caused by some basic level graph optimization. If there is time, some debugging (by disabling basic level graph optimization one by one) can find which optimizer is the cause.
Thanks a lot @tianleiwu
Hi all, is this resolved in 1.17.3 released 2 days ago? @tianleiwu
Describe the issue
Hi, I noticed a regression in
onnxruntime-gpu==1.15.1
andonnxruntime-gpu==1.16.3
(no problem ononnxruntime-gpu==1.14.1
.The following code runs fine on
CPUExecutionProvider
for all three ORT versions, but fails onCUDAExecutionProvider
for1.15.1
and1.16.3
.with the error:
To reproduce
As above. Reproduce with https://huggingface.co/fxmarty/bugged-detr-ort-cuda/tree/main
Using CUDA 11.7, which should be compatible according to https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html
Urgency
medium
Platform
Linux
OS Version
Ubuntu 22.04
ONNX Runtime Installation
Released Package
ONNX Runtime Version or Commit ID
as above
ONNX Runtime API
Python
Architecture
X64
Execution Provider
CUDA
Execution Provider Library Version
CUDA 11.7