search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.85k stars 2.94k forks source link

how to use onnx model to process torch.Tensor data on GPU #22638

Open yangqiheng2019 opened 1 month ago

yangqiheng2019 commented 1 month ago

Describe the issue

Now,I use yolov7 onnx model to process ,but it must to.cpu() ,it means I can not use GPU to process data, it will spend more time. python 3.11.5 torch 2.4.0 onnxruntime 1.18.1

To reproduce

outname = [i.name for i in self.model.get_outputs()] inp = {self.model.get_inputs()[0].name: im.cpu().numpy()} outputs = self.model.run(outname,inp)[0]

Urgency

No response

Platform

Linux

OS Version

Ubuntu 22.04.4

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.18.1

ONNX Runtime API

Python

Architecture

X86

Execution Provider

CUDA

Execution Provider Library Version

12.4

tianleiwu commented 4 weeks ago

You can use I/O binding. See this as example: https://github.com/microsoft/onnxruntime/blob/ba0bb43b00f0d4ef1660acc9f2f581c6c1efdd70/onnxruntime/test/python/onnxruntime_test_python_iobinding.py#L240-L254

Or the last example in https://onnxruntime.ai/docs/api/python/api_summary.html#data-on-device.