Closed sonic182 closed 1 year ago
Yes, that's the correct place to specify the feature flags to enable the execution providers in Ort, however the correct libonnxruntime also needs to be built to support the execution providers you provide. Currently it's only cuda
and tensorrt
because those are available in the precompiled library that is downloaded from Microsoft's releases.
Long term I'd like to have our own precompiled libonnxruntime(s) with more EPs built out, then pull those for the download rather than the Microsoft releases. For now, you'd need to compile onnxruntime yourself and then point to that with ORT_LIB_LOCATION
in addition to adding the feature flag in config.exs
to enable it for Ort.
Ohh I see, I have read ort readme where it says so.
Another question, how can I make ortex to use my nvida gpu? I think it is not using it (nvtop doesn't says the opossite)
I'm running over docker with nvidia-container-runtime, inside the container I see that it is correctly detected the card (with nvidia-smi)
Without knowing the entire setup I can't say for certain why it isn't using your GPU. Are you passing the cuda
option through as the first EP when the model is loaded?
model = Ortex.load("./path/to/model", [:cuda])
If you are and it's still not utilizing the GPU, ensure LD_LIBRARY_PATH
has your cuda version linked and that you have the correct version of cuda and cudnn installed for your driver version.
@mortont it seems that the EP are not enabled in ort
, this is why it's not using the GPU.
I have a Jetson Nano Orin, I compiled onnxruntime
with support for cuda
and tensorrt
. After loading the model I have this output.
Ortex.load("model.onnx", [:tensorrt, :cuda])
The warnings
2023-07-19T19:58:28.807552Z DEBUG ort::environment: Environment not yet initialized, creating a new one
2023-07-19T19:58:28.850619Z DEBUG ort::environment: Environment created env_ptr="0xffff1042c910"
2023-07-19T19:58:28.851381Z WARN apply_execution_providers: ort::execution_providers: `ort` was not compiled to enable TensorrtExecutionProvider - you may be missing a Cargo feature to enable it.
2023-07-19T19:58:28.851403Z WARN apply_execution_providers: ort::execution_providers: `ort` was not compiled to enable CUDAExecutionProvider - you may be missing a Cargo feature to enable it.
2023-07-19T19:58:28.851695Z DEBUG ort: Flush-to-zero and denormal-as-zero are off 2023-07-19T19:58:28.851729Z DEBUG ort: Creating and using per session threadpools since use_per_session_threads_ is true
2023-07-19T19:58:28.851754Z DEBUG ort: Dynamic block base set to 0
...
After enabling: cuda
and tensorrt
in default features in Cargo
[features]
default = ["download-binaries", "cuda", "tensorrt"]
Now everything seems good
2023-07-19T20:03:58.617038Z DEBUG ort::environment: Environment not yet initialized, creating a new one
2023-07-19T20:03:58.654983Z DEBUG ort::environment: Environment created env_ptr="0xfffefc42c910" 2023-07-19T20:03:58.655141Z INFO apply_execution_providers: ort::execution_providers: TensorRT execution provider registered successfully
2023-07-19T20:04:01.109383Z INFO apply_execution_providers: ort::execution_providers: TensorRT execution provider registered successfully
2023-07-19T20:04:01.109456Z INFO apply_execution_providers: ort::execution_providers: TensorRT execution provider registered successfully
2023-07-19T20:04:01.109473Z INFO apply_execution_providers: ort::execution_providers: CUDA execution provider registered successfully
2023-07-19T20:04:01.355042Z INFO apply_execution_providers: ort::execution_providers: CUDA execution provider registered successfully
....
Those feature flags should be enabled here. We don't set them in the cargo features so that elixir can control the features depending on platform.
It sounds like :os.type()
is not returning an expected result on the Jetson Nano Orin, could you run that and let me know what it returns @gBillal?
actually I suspected that at first and so I run that command, however everything is fine
Interesting. I'll see if I can reproduce this on my jetson and work from there.
Something seems off here, when pulling ortex
via hex
there's no config folder in deps/ortex
. So adding a configuration in config.exs
on my project seems to work
import Config
config :ortex, Ortex.Native, features: ["cuda", "tensorrt"]
I can see this
==> ortex
Compiling 6 files (.ex)
Compiling crate ortex in release mode (native/ortex)
["cargo", "rustc", "--features", "cuda,tensorrt", "--release"]
Compiling libc v0.2.139
Compiling cc v1.0.79
Compiling log v0.4.17
Compiling memchr v2.5.0
...
But without the configuration above, this is what I have
==> ortex
Compiling 6 files (.ex)
Compiling crate ortex in release mode (native/ortex)
["cargo", "rustc", "--release"]
Compiling libc v0.2.139
Compiling cc v1.0.79
...
Does this mean we need to add this config in our project. This should be specified in the documentation.
Ah, thank you. Yes, looks like I forgot to include config.exs
in the hex package. I'll add that now. This way, if you want to add different execution providers you can override them in your own config but still have a sane default. I'll add a section in the docs to highlight this as well.
Ortex v0.1.7 is now live on Hex, which includes the default config.exs
as well as some more docs around adding different EPs. I'm going to close this for now, but feel free to reopen if this does not resolve your issue.
There is a way of specify the execution providers for the ort build?
I think you do it here https://github.com/elixir-nx/ortex/blob/841ee6bebd842f43a15b2bb684e4e0691f69a790/config/config.exs#L11C9-L11C9 but I'm not sure.
Would be nice to allow onednn as it allows more performance on AWS arm instances (graviton2, 3...), to allow specifying the providers for compile time (rustler not elixir)