Closed markbarna closed 1 year ago
jkosek
commented
1 year ago Hi @markbarna.
The log you have attached shows the error from the ONNXRuntime + TensorRT accelerator.
Could you share the log from TensorRT runner? Please, search for Correctness test for: Format.TENSORRT <class 'model_navigator.runners.tensorrt.TensorRTRunner'> started. in logs. Thanks!
markbarna
commented
1 year ago @jkosek , thanks for your reply. I actually don't see logs from the TensorRT runner. The full log is:
2023-06-20 19:50:40 INFO Navigator: ============================== Pipeline 'Preprocessing' started ====================================
2023-06-20 19:50:40 INFO Navigator: ============================== Command 'InferInputMetadata' started ================================
2023-06-20 19:50:40 INFO Navigator: Creating ONNX-Runtime Inference Session with providers: ['CUDAExecutionProvider']
2023-06-20 19:50:40.616334516 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:578 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.
2023-06-20 19:50:40 ERROR Navigator: Command finished with unexpected error: Traceback (most recent call last):
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/pipelines/pipeline.py", line 111, in _execute_unit
command_output = execution_unit.command(status).run(
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/commands/base.py", line 78, in run
output = self._run(*args, **_filter_dict_for_func(kwargs, self._run))
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/commands/infer_metadata.py", line 128, in _run
input_names = self._get_default_input_names(model, sample, framework)
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/commands/infer_metadata.py", line 167, in _get_default_input_names
with onnx_runner:
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/runners/base.py", line 146, in __enter__
self.activate()
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/runners/base.py", line 182, in activate
self.activate_impl()
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/runners/onnx.py", line 122, in activate_impl
self.sess, _ = utils.invoke_if_callable(self._sess)
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/utils/common.py", line 312, in invoke_if_callable
ret = func(*args, **kwargs)
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/runners/onnx.py", line 59, in __call__
return self.call_impl(*args, **kwargs)
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/model_navigator/runners/onnx.py", line 82, in call_impl
return onnxrt.InferenceSession(model_bytes, providers=providers)
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 347, in __init__
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/workspace/virtualenvs/model-navigator/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 395, in _create_inference_session
sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for EfficientNMS_TRT(1) node with name 'batchedNMS'Also, for reference, my script looks like:
import model_navigator as nav
from pathlib import Path
import cv2
import numpy as np
def resize_image(frame: np.ndarray, target_width=640, target_height=360) -> np.ndarray:
"""Resize the image for the frame
Args:
frame (np.ndarray): the image frame
Returns:
frame: Resize the image frames
"""
if target_width is not None and target_height is not None:
frame = cv2.resize(
frame,
dsize=(target_width, target_height),
interpolation=cv2.INTER_LINEAR,
)
return frame
def frames_preprocessing(frames: np.ndarray, **kwargs) -> dict:
# Define processed frames
processed_frames = []
# Do all the steps for the fram
for frame in frames:
# Color the space from BGR to RGB
processed_frame = frame[:, :, ::-1]
# Downsize the camera image
processed_frame = resize_image(processed_frame)
# Pad the frame
processed_frame = np.pad(
processed_frame,
pad_width=((12, 12), (0, 0), (0, 0)),
constant_values=114,
)
# reorder axes to Channel x Height x Width
processed_frame = np.moveaxis(processed_frame, -1, 0)
# normalize
processed_frame = processed_frame / 255
processed_frame = np.expand_dims(processed_frame, 0)
# Add this to frames
processed_frames.append(processed_frame)
# Return something that is a modified version of whatever comes in here
return processed_frames
if __name__ == "__main__":
input_dir = Path("<DIRECTORY WITH INPUT ONNX MODEL>")
images_path = Path("<SAMPLES IMAGES DIRECTORY>")
# load data
images = [cv2.imread(p.as_posix()) for p in images_path.iterdir()]
dataset = frames_preprocessing(images)
dataset = [img.astype("float32") for img in dataset]
config = {
"custom_configs": [
nav.TensorRTConfig(
trt_profile=nav.TensorRTProfile().add("images", min=(1, 3, 384, 640), max=(16, 3, 384, 640), opt=(16, 3, 384, 640))
)
],
"profiler_config": nav.ProfilerConfig(run_profiling=True, batch_sizes=[1, 2, 4, 8, 16]),
}
package = nav.onnx.optimize(
model=input_dir / "path to model.onnx",
dataloader=dataset,
workspace=input_dir / "workspace",
**config
)
nav.package.save(package, input_dir / "output_package", override=True)Thanks for your help!
jkosek
commented
1 year ago @markbarna the model you are using is already a modified ONNX model that cannot be executed on default ONNX runtimes. The Model Navigator perform following steps on input model:
When you provide a modified file this basic operations are unable to be executed on default ONNX runners. The only valid path is to directly convert the model to TensorRT. At this point we are not able to handle your ONNX model.
Could you provide following info:
We would like to review options how we could handle such case. Thanks!
markbarna
commented
1 year ago @jkosek , thanks for looking into it. Here is the info you requested:
EfficientNMS_TRT operator in place of this client-side operation: https://github.com/ultralytics/yolov5/blob/master/utils/general.py#L856 Thank you
jkosek
commented
1 year ago @markbarna thanks for sharing more details. In TenosrRT 8.6 version the native NonMaxSuppression operator will map to new INMSLayer. Seems like applying custom operation is not necessary. You can also read more about plugin limitations here: https://github.com/NVIDIA/TensorRT/tree/release/8.6/plugin/efficientNMSPlugin#limitations
In Model Navigator we are not supporting ONNX models with TensorRT custom nodes embedded in graph at the moment. If you still need to modify the ONNX graph my suggestion would be to use trtexec in such cases.
github-actions[bot]
commented
1 year ago This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.
github-actions[bot]
commented
1 year ago This issue was closed because it has been stalled for 7 days with no activity.
I have an object detection model that was exported to ONNX and then had this http://www.xavierdupre.fr/app/onnxcustom/helpsphinx/api/onnxops/onnx__EfficientNMS_TRT.html attached using onnx_graphsurgeon.
In the past I've used
trtexecto convert the model to TensorRT. This succeeds without any issues, but I'm trying to use model navigator to convert the same model and get the following error. Is there a way to tell it how to load this plugin?