Ort (onnx) execution providers

[sonic182]

There is a way of specify the execution providers for the ort build?

I think you do it here https://github.com/elixir-nx/ortex/blob/841ee6bebd842f43a15b2bb684e4e0691f69a790/config/config.exs#L11C9-L11C9 but I'm not sure.

Would be nice to allow onednn as it allows more performance on AWS arm instances (graviton2, 3...), to allow specifying the providers for compile time (rustler not elixir)

[mortont]

Yes, that's the correct place to specify the feature flags to enable the execution providers in Ort, however the correct libonnxruntime also needs to be built to support the execution providers you provide. Currently it's only cuda and tensorrt because those are available in the precompiled library that is downloaded from Microsoft's releases.

Long term I'd like to have our own precompiled libonnxruntime(s) with more EPs built out, then pull those for the download rather than the Microsoft releases. For now, you'd need to compile onnxruntime yourself and then point to that with ORT_LIB_LOCATION in addition to adding the feature flag in config.exs to enable it for Ort.

[sonic182]

Ohh I see, I have read ort readme where it says so.

Another question, how can I make ortex to use my nvida gpu? I think it is not using it (nvtop doesn't says the opossite)

I'm running over docker with nvidia-container-runtime, inside the container I see that it is correctly detected the card (with nvidia-smi)

[mortont]

Without knowing the entire setup I can't say for certain why it isn't using your GPU. Are you passing the cuda option through as the first EP when the model is loaded?

model = Ortex.load("./path/to/model", [:cuda])

If you are and it's still not utilizing the GPU, ensure LD_LIBRARY_PATH has your cuda version linked and that you have the correct version of cuda and cudnn installed for your driver version.

[gBillal]

@mortont it seems that the EP are not enabled in ort, this is why it's not using the GPU. I have a Jetson Nano Orin, I compiled onnxruntime with support for cuda and tensorrt. After loading the model I have this output.

Ortex.load("model.onnx", [:tensorrt, :cuda])

The warnings

2023-07-19T19:58:28.807552Z DEBUG ort::environment: Environment not yet initialized, creating a new one
2023-07-19T19:58:28.850619Z DEBUG ort::environment: Environment created env_ptr="0xffff1042c910"
2023-07-19T19:58:28.851381Z  WARN apply_execution_providers: ort::execution_providers: `ort` was not compiled to enable TensorrtExecutionProvider - you may be missing a Cargo feature to enable it.
2023-07-19T19:58:28.851403Z  WARN apply_execution_providers: ort::execution_providers: `ort` was not compiled to enable CUDAExecutionProvider - you may be missing a Cargo feature to enable it.
2023-07-19T19:58:28.851695Z DEBUG ort: Flush-to-zero and denormal-as-zero are off                                                                                                                                                                                                      2023-07-19T19:58:28.851729Z DEBUG ort: Creating and using per session threadpools since use_per_session_threads_ is true
2023-07-19T19:58:28.851754Z DEBUG ort: Dynamic block base set to 0
...

After enabling: cuda and tensorrt in default features in Cargo

[features]
default = ["download-binaries", "cuda", "tensorrt"]

Now everything seems good

2023-07-19T20:03:58.617038Z DEBUG ort::environment: Environment not yet initialized, creating a new one
2023-07-19T20:03:58.654983Z DEBUG ort::environment: Environment created env_ptr="0xfffefc42c910"                                                                                                                                                                                                       2023-07-19T20:03:58.655141Z  INFO apply_execution_providers: ort::execution_providers: TensorRT execution provider registered successfully
2023-07-19T20:04:01.109383Z  INFO apply_execution_providers: ort::execution_providers: TensorRT execution provider registered successfully
2023-07-19T20:04:01.109456Z  INFO apply_execution_providers: ort::execution_providers: TensorRT execution provider registered successfully
2023-07-19T20:04:01.109473Z  INFO apply_execution_providers: ort::execution_providers: CUDA execution provider registered successfully
2023-07-19T20:04:01.355042Z  INFO apply_execution_providers: ort::execution_providers: CUDA execution provider registered successfully
....

[mortont]

Those feature flags should be enabled here. We don't set them in the cargo features so that elixir can control the features depending on platform.

It sounds like :os.type() is not returning an expected result on the Jetson Nano Orin, could you run that and let me know what it returns @gBillal?

[gBillal]

actually I suspected that at first and so I run that command, however everything is fine

[mortont]

Interesting. I'll see if I can reproduce this on my jetson and work from there.

[gBillal]

Something seems off here, when pulling ortex via hex there's no config folder in deps/ortex. So adding a configuration in config.exs on my project seems to work

import Config

config :ortex, Ortex.Native, features: ["cuda", "tensorrt"]

I can see this

==> ortex
Compiling 6 files (.ex)
Compiling crate ortex in release mode (native/ortex)
["cargo", "rustc", "--features", "cuda,tensorrt", "--release"]
   Compiling libc v0.2.139
   Compiling cc v1.0.79
   Compiling log v0.4.17
   Compiling memchr v2.5.0
...

But without the configuration above, this is what I have

==> ortex
Compiling 6 files (.ex)
Compiling crate ortex in release mode (native/ortex)
["cargo", "rustc", "--release"]
   Compiling libc v0.2.139
   Compiling cc v1.0.79
...

Does this mean we need to add this config in our project. This should be specified in the documentation.

[mortont]

Ah, thank you. Yes, looks like I forgot to include config.exs in the hex package. I'll add that now. This way, if you want to add different execution providers you can override them in your own config but still have a sane default. I'll add a section in the docs to highlight this as well.

[mortont]

Ortex v0.1.7 is now live on Hex, which includes the default config.exs as well as some more docs around adding different EPs. I'm going to close this for now, but feel free to reopen if this does not resolve your issue.