search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.15k stars 2.85k forks source link

can we customize memory allocation functions(like malloc/free) for inference in C api? #16032

Open dandlake opened 1 year ago

dandlake commented 1 year ago

Describe the issue

When using C api to do inference, I want to customize the memory allocation functions like malloc/free to control how onnxruntime allocates memory. I want to make onnxruntime use a separate area of memory in some shared memory region. Is there a way to do this?

I looked at the inference example with C in https://github.com/microsoft/onnxruntime-inference-examples/blob/main/c_cxx/fns_candy_style_transfer/fns_candy_style_transfer.c

I also looked at the following page: https://onnxruntime.ai/docs/get-started/with-c.html

I see there is some custom allocator support, but looks like they are for other purposes.

To reproduce

N/A

Urgency

No response

Platform

Linux

OS Version

ubuntu

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.14.1

ONNX Runtime API

C

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

pranavsharma commented 1 year ago

You can build ORT with mimalloc: https://github.com/microsoft/onnxruntime/blob/76fd9aa745db596d5a1a8053fa76a1f0c92cd1b7/tools/ci_build/build.py#L468.