TFLite, ONNX, CoreML, TensorRT Export

glenn-jocher commented 4 years ago

📚 This guide explains how to export a trained YOLOv5 🚀 model from PyTorch to ONNX and TorchScript formats. UPDATED 8 December 2022.

Before You Start

Clone repo and install requirements.txt in a Python>=3.7.0 environment, including PyTorch>=1.7. Models and datasets download automatically from the latest YOLOv5 release.

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

For TensorRT export example (requires GPU) see our Colab notebook appendix section.

Formats

YOLOv5 inference is officially supported in 11 formats:

💡 ProTip: Export to ONNX or OpenVINO for up to 3x CPU speedup. See CPU Benchmarks. 💡 ProTip: Export to TensorRT for up to 5x GPU speedup. See GPU Benchmarks.

Format	`export.py --include`	Model
PyTorch	-	`yolov5s.pt`
TorchScript	`torchscript`	`yolov5s.torchscript`
ONNX	`onnx`	`yolov5s.onnx`
OpenVINO	`openvino`	`yolov5s_openvino_model/`
TensorRT	`engine`	`yolov5s.engine`
CoreML	`coreml`	`yolov5s.mlmodel`
TensorFlow SavedModel	`saved_model`	`yolov5s_saved_model/`
TensorFlow GraphDef	`pb`	`yolov5s.pb`
TensorFlow Lite	`tflite`	`yolov5s.tflite`
TensorFlow Edge TPU	`edgetpu`	`yolov5s_edgetpu.tflite`
TensorFlow.js	`tfjs`	`yolov5s_web_model/`
PaddlePaddle	`paddle`	`yolov5s_paddle_model/`

Benchmarks

Benchmarks below run on a Colab Pro with the YOLOv5 tutorial notebook . To reproduce:

python benchmarks.py --weights yolov5s.pt --imgsz 640 --device 0

Colab Pro V100 GPU

benchmarks: weights=/content/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml, device=0, half=False, test=False
Checking setup...
YOLOv5 🚀 v6.1-135-g7926afc torch 1.10.0+cu111 CUDA:0 (Tesla V100-SXM2-16GB, 16160MiB)
Setup complete ✅ (8 CPUs, 51.0 GB RAM, 46.7/166.8 GB disk)

Benchmarks complete (458.07s)
                   Format  mAP@0.5:0.95  Inference time (ms)
0                 PyTorch        0.4623                10.19
1             TorchScript        0.4623                 6.85
2                    ONNX        0.4623                14.63
3                OpenVINO           NaN                  NaN
4                TensorRT        0.4617                 1.89
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel        0.4623                21.28
7     TensorFlow GraphDef        0.4623                21.22
8         TensorFlow Lite           NaN                  NaN
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Colab Pro CPU

benchmarks: weights=/content/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml, device=cpu, half=False, test=False
Checking setup...
YOLOv5 🚀 v6.1-135-g7926afc torch 1.10.0+cu111 CPU
Setup complete ✅ (8 CPUs, 51.0 GB RAM, 41.5/166.8 GB disk)

Benchmarks complete (241.20s)
                   Format  mAP@0.5:0.95  Inference time (ms)
0                 PyTorch        0.4623               127.61
1             TorchScript        0.4623               131.23
2                    ONNX        0.4623                69.34
3                OpenVINO        0.4623                66.52
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel        0.4623               123.79
7     TensorFlow GraphDef        0.4623               121.57
8         TensorFlow Lite        0.4623               316.61
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Export a Trained YOLOv5 Model

This command exports a pretrained YOLOv5s model to TorchScript and ONNX formats. yolov5s.pt is the 'small' model, the second smallest model available. Other options are yolov5n.pt, yolov5m.pt, yolov5l.pt and yolov5x.pt, along with their P6 counterparts i.e. yolov5s6.pt or you own custom training checkpoint i.e. runs/exp/weights/best.pt. For details on all available models please see our README table.

python export.py --weights yolov5s.pt --include torchscript onnx

💡 ProTip: Add --half to export models at FP16 half precision for smaller file sizes

Output:

export: data=data/coco128.yaml, weights=['yolov5s.pt'], imgsz=[640, 640], batch_size=1, device=cpu, half=False, inplace=False, train=False, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=12, verbose=False, workspace=4, nms=False, agnostic_nms=False, topk_per_class=100, topk_all=100, iou_thres=0.45, conf_thres=0.25, include=['torchscript', 'onnx']
YOLOv5 🚀 v6.2-104-ge3e5122 Python-3.7.13 torch-1.12.1+cu113 CPU

Downloading https://github.com/ultralytics/yolov5/releases/download/v6.2/yolov5s.pt to yolov5s.pt...
100% 14.1M/14.1M [00:00<00:00, 274MB/s]

Fusing layers... 
YOLOv5s summary: 213 layers, 7225885 parameters, 0 gradients

PyTorch: starting from yolov5s.pt with output shape (1, 25200, 85) (14.1 MB)

TorchScript: starting export with torch 1.12.1+cu113...
TorchScript: export success ✅ 1.7s, saved as yolov5s.torchscript (28.1 MB)

ONNX: starting export with onnx 1.12.0...
ONNX: export success ✅ 2.3s, saved as yolov5s.onnx (28.0 MB)

Export complete (5.5s)
Results saved to /content/yolov5
Detect:          python detect.py --weights yolov5s.onnx 
Validate:        python val.py --weights yolov5s.onnx 
PyTorch Hub:     model = torch.hub.load('ultralytics/yolov5', 'custom', 'yolov5s.onnx')
Visualize:       https://netron.app/

The 3 exported models will be saved alongside the original PyTorch model:

Netron Viewer is recommended for visualizing exported models:

Exported Model Usage Examples

detect.py runs inference on exported models:

python detect.py --weights yolov5s.pt                 # PyTorch
                           yolov5s.torchscript        # TorchScript
                           yolov5s.onnx               # ONNX Runtime or OpenCV DNN with --dnn
                           yolov5s_openvino_model     # OpenVINO
                           yolov5s.engine             # TensorRT
                           yolov5s.mlmodel            # CoreML (macOS only)
                           yolov5s_saved_model        # TensorFlow SavedModel
                           yolov5s.pb                 # TensorFlow GraphDef
                           yolov5s.tflite             # TensorFlow Lite
                           yolov5s_edgetpu.tflite     # TensorFlow Edge TPU
                           yolov5s_paddle_model       # PaddlePaddle

val.py runs validation on exported models:

python val.py --weights yolov5s.pt                 # PyTorch
                        yolov5s.torchscript        # TorchScript
                        yolov5s.onnx               # ONNX Runtime or OpenCV DNN with --dnn
                        yolov5s_openvino_model     # OpenVINO
                        yolov5s.engine             # TensorRT
                        yolov5s.mlmodel            # CoreML (macOS Only)
                        yolov5s_saved_model        # TensorFlow SavedModel
                        yolov5s.pb                 # TensorFlow GraphDef
                        yolov5s.tflite             # TensorFlow Lite
                        yolov5s_edgetpu.tflite     # TensorFlow Edge TPU
                        yolov5s_paddle_model       # PaddlePaddle

Use PyTorch Hub with exported YOLOv5 models:

import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'custom', 'yolov5s.pt')
                                                       'yolov5s.torchscript ')       # TorchScript
                                                       'yolov5s.onnx')               # ONNX Runtime
                                                       'yolov5s_openvino_model')     # OpenVINO
                                                       'yolov5s.engine')             # TensorRT
                                                       'yolov5s.mlmodel')            # CoreML (macOS Only)
                                                       'yolov5s_saved_model')        # TensorFlow SavedModel
                                                       'yolov5s.pb')                 # TensorFlow GraphDef
                                                       'yolov5s.tflite')             # TensorFlow Lite
                                                       'yolov5s_edgetpu.tflite')     # TensorFlow Edge TPU
                                                       'yolov5s_paddle_model')       # PaddlePaddle

# Images
img = 'https://ultralytics.com/images/zidane.jpg'  # or file, Path, PIL, OpenCV, numpy, list

# Inference
results = model(img)

# Results
results.print()  # or .show(), .save(), .crop(), .pandas(), etc.

OpenCV DNN inference

OpenCV inference with ONNX models:

python export.py --weights yolov5s.pt --include onnx

python detect.py --weights yolov5s.onnx --dnn  # detect
python val.py --weights yolov5s.onnx --dnn  # validate

C++ Inference

YOLOv5 OpenCV DNN C++ inference on exported ONNX model examples:

YOLOv5 OpenVINO C++ inference examples:

TensorFlow.js Web Browser Inference

https://aukerul-shuvo.github.io/YOLOv5_TensorFlow-JS/

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Notebooks with free GPU:
Google Cloud Deep Learning VM. See GCP Quickstart Guide
Amazon Deep Learning AMI. See AWS Quickstart Guide
Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

dsuess commented 4 years ago

@aisensiy It's a bit confusing... The tag v2.0 doesn't seem to point to the v2.0 models. Check out these changes, I've just tested and confirmed that this works: https://github.com/dsuess/yolov5/commit/a68162c37fbe624a8f2ce63bf4e16d5c8c4de06e

glenn-jocher commented 4 years ago

@aisensiy @dsuess see https://github.com/ultralytics/yolov5/issues/831 for a recently implemented export fix. Full export functionality is now available for the most recent v3.0 YOLOv5 models, as well as previous version models (including pytorch 1.5 trained models, v2.0 models, etc) to all destinations (torchscript, onnx and coreml). A new model fusion feature addition has been added as well (https://github.com/ultralytics/yolov5/issues/827).

Please git pull to get the most recent updates, or simply reclone the repo and try again.

aisensiy commented 4 years ago

Wow, soooo fast reply. I tried the latest commit. Exporing works!

nobody-cheng commented 4 years ago

version
Python 3.7.8
torch 1.6.0+cu101
onnx 1.7.0

TorchScript export success, saved as ./runs/exp0/weights/best.torchscript.pt

Starting ONNX export with onnx 1.7.0...
Fusing layers...
Model Summary: 236 layers, 4.737e+07 parameters, 4.48868e+07 gradients
ONNX export failure: Exporting the operator hardswish to ONNX opset version 12 is not supported. Please open a bug to request ONNX export support for the missing operator.

Starting CoreML export with coremltools 3.4...
CoreML export failure: module 'coremltools' has no attribute 'convert'

yuanyuangoo commented 4 years ago

Hi, is there any way to export onnx or torchscript enable dynamic input shape?

glenn-jocher commented 4 years ago

@yuanyuangoo I don't know, that's a good question. CoreML allows for this after export using coremltools, but I'm not sure about the other two. https://apple.github.io/coremltools/generated/coremltools.models.neural_network.flexible_shape_utils.html

MohamedAliRashad commented 4 years ago

@glenn-jocher I get this error AttributeError: 'Detect' object has no attribute 'm' with PyTorch==1.5.1 & torchvision==0.6.1 & onnx==1.7

Tried PyTorch==1.6.0 but also failed with the same error.
Changed from Hardswish to LeakRelu in PyTorch 1.5.1
added the yolov5 path to PYTHONPATH

Any ideas ?

jepetolee commented 4 years ago

there are three scenarios, First, the pt file was damaged, use other pt files. Second, change argpase on detect.py to absolute path, Third, your python code was't updated so, download project again, and try again. if you use kernel, I prefer you to use pycharm, change codes and runs on the pycharm IDE.

MohamedAliRashad commented 4 years ago

The pt is good, i tested it multiple times
you mean export.py, did that and didn't work too
have the latest version of the Repo

hyperparameters commented 4 years ago

Model traacing is incorrect

torch==1.6

device="cuda"
imgsz=640
img = torch.zeros((1, 3, imgsz, imgsz), device=device)

The output shape after loading model in models/export.py

model = torch.load("./runs/exp32/weights/last.pt", map_location=torch.device('cpu'))['model'].float()
model.eval()
model.model[-1].export = True  # set Detect() layer export=True
y = model(img)  # dry run
print(y[0].shape)
>>> torch.Size([1, 3, 80, 80, 6])

The output shape when loaded in detect.py

model = attempt_load("./runs/exp32/weights/last.pt",device)
model.eval();
y = model(img)
print(y[0].shape)
>>> torch.Size([1, 25200, 6])

output when tracing

TracerWarning: Encountering a list at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the module's inputs. Consider using a constant container instead (e.g. for `list`, use a `tuple` instead. for `dict`, use a `NamedTuple` instead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior.
  module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace)
TorchScript export success, saved as yolov5s.torchscript.pt

pranavravella commented 4 years ago

I'm not sure if this is a AWS issue or tracing issue. I'm currently using Yolov5x in an ONNX format to be compiled for an ARM processor using the AWS Neo cross-compiler. Given an input data configuration of {"images":[1, 3, 640, 640]} to the Neo compilation job, it is stated that "TVM cannot convert ONNX model, and relay.concatenate requires all tensors have the same shape on non-concatenating axes". The Neo compilation worked perfectly fine on the yolov3-spp ONNX model. My guess that it is something in here, as mentioned before, but I cannot place the issue.

img = torch.zeros((opt.batch_size, 3, *opt.img_size))  # image size(1,3,320,192) iDetection
    # Load PyTorch model
    model = attempt_load(opt.weights, map_location=torch.device('cpu'))  # load FP32 model

    # Update model
    for k, m in model.named_modules():
        m._non_persistent_buffers_set = set()  # pytorch 1.6.0 compatability
        if isinstance(m, Conv) and isinstance(m.act, nn.Hardswish):
            m.act = Hardswish()  # assign activation
        # if isinstance(m, Detect):
        #    m.forward = m.forward_export  # assign forward (optional)
    model.model[-1].export = True  # set Detect() layer export=True

The error occurs when the input configuration is wrong ({"images":[1, 3, 640, 640]}), am I missing something or is this a bug? I can provide a more detailed error description if necessary.

xiaosongshine commented 4 years ago

Would CoreML failure as shown below affect the successfully converted onnx model? Thank you.

ONNX export success, saved as weights/yolov5s.onnx WARNING:root:TensorFlow version 2.2.0 detected. Last version known to be fully compatible is 1.14.0 . WARNING:root:Keras version 2.4.3 detected. Last version known to be fully compatible of Keras is 2.2.4 .

Starting CoreML export with coremltools 3.4... CoreML export failure: module 'coremltools' has no attribute 'convert'

Export complete. Visualize with https://github.com/lutzroeder/netron fix this error by: pip install coremltools==4.0b3 pip install packaging

glenn-jocher commented 4 years ago

@xiaosongshine no, they are independent. Yes, your fix looks correct. See requirements.txt for a full list of dependencies.

pranavravella commented 4 years ago

Is there any update on the issue I presented earlier in the thread? Sorry for bothering you all!

luvwinnie commented 4 years ago

is this possible to create with variable batch size onnx model?

nobody-cheng commented 4 years ago

is this possible to create with variable batch size onnx model?

torch.onnx.export(self.model,  # model being run
inputs,  # model input (or a tuple for multiple inputs)
export_onnx_file,  # where to save the model (can be a file or file-like object)
export_params=True,  # store the trained parameter weights inside the model file
opset_version=10,  # the ONNX version to export the model to
do_constant_folding=True,  # whether to execute constant folding for optimization
input_names=['input'],  # the model's input names
output_names=['output'],  # the model's output names
dynamic_axes={'input': {0: 'batch_size'},  # variable lenght axes  # 批处理
'output': {0: 'batch_size'}})

satheeshkatipomu commented 4 years ago

@nobody-cheng, @luvwinnie , dynamic batch size is working for you?. I tried doing the same.I used batch_size=16 while exporting, trying to infer on batch_size=32. But I am getting the following error at the time of inference.

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_931' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:43 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, std::vector<long int>&) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false.
The input tensor cannot be reshaped to the requested shape. Input shape:{32,3,64,64,2}, requested shape:{16,3,64,64,2}

nobody-cheng commented 4 years ago

@nobody-cheng, @luvwinnie , dynamic batch size is working for you?. I tried doing the same.I used batch_size=16 while exporting, trying to infer on batch_size=32. But I am getting the following error at the time of inference.

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_931' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:43 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, std::vector<long int>&) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false.
The input tensor cannot be reshaped to the requested shape. Input shape:{32,3,64,64,2}, requested shape:{16,3,64,64,2}

for me personally is work

def inference(devices, img_paths, index, batchsize):
os.environ['CUDA_VISIBLE_DEVICES'] = str(devices)
features = list()
ort_sess = onnxruntime.InferenceSession(args.model_path)
input_name = ort_sess.get_inputs()[0].name
images = list()
for img_path in img_paths:
image = preprocess(img_path, args.height, args.width)
if len(images) < batchsize:
images.append(image)
continue
else:
intput_image = np.concatenate(images)
feat = ort_sess.run(None, {input_name: intput_image})[0]
feat = normalize(feat, axis=1)
for i in feat:
features.append(i)
images.clear()

satheeshkatipomu commented 4 years ago

Thank you for the response. which of the model configs(yolov5[s,l,m,x]) are you using?

zgh2022 commented 4 years ago

In running models/export.py, I meet a mistake on export my own .pt(base in yolov5s) to onnx: Traceback (most recent call last): File "models/export.py", line 41, in y = model(img) # dry run File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, *kwargs) File "/content/yolov5/models/yolo.py", line 113, in forward return self.forward_once(x, profile) # single-scale inference, train File "/content/yolov5/models/yolo.py", line 133, in forward_once x = m(x) # run File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, **kwargs) File "/content/yolov5/models/common.py", line 98, in forward return torch.cat(x, self.d) RuntimeError: Sizes of tensors must match except in dimension 1. Got 20 and 19 in dimension 2 (The offending index is 1)

How can I solve this mistake?

zgh2022 commented 4 years ago

In running models/export.py, I meet a mistake on export my own .pt(base in yolov5s) to onnx: Traceback (most recent call last): File "models/export.py", line 41, in y = model(img) # dry run File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, *kwargs) File "/content/yolov5/models/yolo.py", line 113, in forward return self.forward_once(x, profile) # single-scale inference, train File "/content/yolov5/models/yolo.py", line 133, in forward_once x = m(x) # run File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, **kwargs) File "/content/yolov5/models/common.py", line 98, in forward return torch.cat(x, self.d) RuntimeError: Sizes of tensors must match except in dimension 1. Got 20 and 19 in dimension 2 (The offending index is 1)

How can I solve this mistake?

Oh，I solve it，because my image size set in onnx-export is different from size in .pt. But I also have a new question: How do I set image size with 300*300 on yolov5?

nobody-cheng commented 4 years ago

Thank you for the response. which of the model configs(yolov5[s,l,m,x]) are you using?

yolov5l torch==1.6.0 torchvision==0.7.0 onnx 1.7.0 only use best.onnx

Namespace(batch_size=1, img_size=[640, 640], weights='./runs/exp6/weights/best.pt')
Fusing layers...
Model Summary: 236 layers, 4.74401e+07 parameters, 4.48868e+07 gradients

Starting TorchScript export with torch 1.6.0...
/home/ubuntu/anaconda3/envs/cz/lib/python3.7/site-packages/torch/jit/__init__.py:1109: TracerWarning: Encountering a list at the output of the tracer might cause the trace to be incorrecte does not change based on the module's inputs. Consider using a constant container instead (e.g. for `list`, use a `tuple` instead. for `dict`, use a `NamedTuple` instead). If you absolu strict=False to trace() to allow this behavior.
  module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace)
TorchScript export success, saved as ./runs/exp6/weights/best.torchscript.pt

Starting ONNX export with onnx 1.7.0...
ONNX export success, saved as ./runs/exp6/weights/best.onnx

Starting CoreML export with coremltools 4.0b2...
CoreML export failure: module 'torch._C' has no attribute '_jit_pass_canonicalize_ops'

Export complete. Visualize with https://github.com/lutzroeder/netron.

blackHorz commented 4 years ago

I am trying to generate torchscript file. Here is what i did so far. OS : Windows torch : 1.6.0 model: yolov5l

I have trained yolov5 on custom dataset and saved the weight as best.pt

Now I am encountering 2 problems

Running this command does not helps python models/export.py --weights runs/exp9/weights/best.pt --img 416 --batch 1

Error : File "models/export.py", line 10, in import models ModuleNotFoundError: No module named 'models'

This behaviour is strange because i can still import module from python. Anyway I moved the script from models/export.py to base directory.

Now it runs the export.py file : new command python export.py --weights runs/exp9/weights/best.pt --img 416 --batch 1

log trace : Namespace(batch_size=1, img_size=[416, 416], weights='runs/exp9/weights/best.pt') Fusing layers... Model Summary: 236 layers, 4.73808e+07 parameters, 4.48868e+07 gradients

Starting TorchScript export with torch 1.6.0... D:\yolov5-master\yolov5-master\models\yolo.py:45: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if self.grid[i].shape[2:4] != x[i].shape[2:4]: C:\Users\st34616\Anaconda3\envs\py38\lib\site-packages\torch\jit__init__.py:1109: TracerWarning: Encountering a list at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the module's inputs. Consider using a constant container instead (e.g. for list, use a tuple instead. for dict, use a NamedTuple instead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior. module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace) TorchScript export success, saved as runs/exp9/weights/best.torchscript.pt ONNX export failure: No module named 'onnx' CoreML export failure: No module named 'coremltools'

Export complete. Visualize with https://github.com/lutzroeder/netron.

Again why it exports to best.torchscript.pt format but not just *.torchscript?

Ultraopxt commented 4 years ago

I am try to run : !python models/export.py --weights yolov5s.pt --img 640 --batch 1 # export

and got an error:

Traceback (most recent call last): File "models/export.py", line 12, in import models ModuleNotFoundError: No module named 'models'

how can i slove it?thanks

nobody-cheng commented 4 years ago

I am try to run : !python models/export.py --weights yolov5s.pt --img 640 --batch 1 # export

and got an error:

Traceback (most recent call last): File "models/export.py", line 12, in import models ModuleNotFoundError: No module named 'models'

how can i slove it?thanks
cd yolov5
export PYTHONPATH="$PWD"  # ******
python models/export.py --weights yolov5s.pt --img 640 --batch 1

Edwardmark commented 4 years ago

@glenn-jocher how to export model trained with sync-bn, I trained the model with SyncBatchNorm. When I try to convert the model to onnx using the default configuration, I encountered the following error:

So I try to convert the model with gpu, I set device to 'cuda', but met another error: ONNX export failure: All input tensors must be on the same device. Received cuda:0 and cpu

https://github.com/ultralytics/yolov5/issues/1018

Thanks.

Edwardmark commented 4 years ago

I am try to run : !python models/export.py --weights yolov5s.pt --img 640 --batch 1 # export and got an error: Traceback (most recent call last): File "models/export.py", line 12, in import models ModuleNotFoundError: No module named 'models' how can i slove it?thanks
cd yolov5
export PYTHONPATH="$PWD"  # ******
python models/export.py --weights yolov5s.pt --img 640 --batch 1

you should add the yolo-v5 directory into python path.

molyswu commented 4 years ago

Edwardmark , You tried to conversion with latest models.

Edwardmark commented 4 years ago

@glenn-jocher Hi, how to export the Detect to onnx as well? I tried the export.py, the onxx output do not contain detect but with three output.

EvgeniiTitov commented 4 years ago

Hi everyone,

Apologies for the stupid question I am about to ask (I am new to C++).

I've jitted my trained model following the pipeline described above. Now, when I am attempting to run it from C++ I am getting a weird Debug Error!

Could anyone please explain why this might be happening? Interestingly, the default ResNet model from the torchvision library jitted and run in the same way works like a charm. Here is the C++ code I use to run the model:

Thanks in advance. Regards, E

pfeatherstone commented 4 years ago

Is it necessary to have fixed input size when exporting? I can understand there may be issues with setting the grid offsets at run time with dynamic input, but if the JIT tools are used, can't that be dealt with?

Alex866850 commented 4 years ago

I am try to run : !python models/export.py --weights yolov5s.pt --img 640 --batch 1 # export

and got an error:

Traceback (most recent call last): File "models/export.py", line 12, in import models ModuleNotFoundError: No module named 'models'

how can i slove it?thanks

把export.py放到根目录文件下

zhiqwang commented 4 years ago

I am try to run : !python models/export.py --weights yolov5s.pt --img 640 --batch 1 # export

and got an error:

Traceback (most recent call last): File "models/export.py", line 12, in import models ModuleNotFoundError: No module named 'models'

how can i slove it?thanks

!python -m models.export --weights yolov5s.pt --img 640 --batch 1

Use this instead can also address this issue.

ltm920716 commented 4 years ago

hello @glenn-jocher ， my env is python3.6， pytorch==1.6.0，onnx=1.7.0，tensorrt==7，cuda10.2，2020TI. I run ‘python models/export.py --weights weights/yolov5s.pt --img 640 --batch 1’， then I make onnx to trt，I get the error： please help！

glenn-jocher commented 4 years ago

@ltm920716 see TensorRT Deployment tutorial: https://docs.ultralytics.com/yolov5

yoni-f commented 4 years ago

Quick question - doesn't the number of bounding boxes depend on the image under inference? In the exported ONNX model, the output has a fixed shape of [(1, 3, 40, 40, 85), (1, 3, 20, 20, 85), (1, 3, 10, 10, 85)], which can be concated together to [(1, 6300, 85)]. But when I look at the test.py, the bus.jpg image produces an output of [(1, 10584, 85)].

Does this mean that the number of bounding boxes predicted in the export models is blocked at 1050 (6300/6)? Any effect on accuracy?

zhiqwang commented 4 years ago

The number of bounding boxes is sensitive to the image shape passed in the network.

yoni-f commented 4 years ago

I know, but how does that sit with the fact that in TRT the graph output is fixed? Or you mean that every image of size 640x640 will have the same number of bounding boxes?

glenn-jocher commented 4 years ago

@yoni-f there will be 3 boxes at each grid point for each yolo output layer with strides 8, 16, 32. Your output size is not related to anyone else's output size, it is only correlated to your input image size.

pfeatherstone commented 4 years ago

I've been trying to get a yolo model to be ONNX exportable with dynamic input size, while not having to manually add grid offsets or apply anchors. I have the following:

@torch.jit.script
def yolo_predict(pred, anchors, stride: int, scale_x_y: float):
    #transpose
    nA = anchors.size(0)
    nB, nC, nH, nW = pred.size()
    nattr = int(nC / nA)
    pred = pred.view(nB, nA, nattr, nH, nW).permute(0, 1, 3, 4, 2).contiguous()

    #make grid
    grid_x = torch.arange(nW).view(1, 1, 1, nW)
    grid_y = torch.arange(nH).view(1, 1, nH, 1)
    anchor_w = anchors[:, 0].view(1, nA, 1, 1).float() / (nW * stride)
    anchor_h = anchors[:, 1].view(1, nA, 1, 1).float() / (nH * stride)

    #apply grid
    x = pred[..., 0]
    y = pred[..., 1]
    w = pred[..., 2]
    h = pred[..., 3]
    p = pred[..., 4:]
    bx = ((torch.sigmoid(x) - 0.5) * scale_x_y + 0.5 + grid_x) / nW
    by = ((torch.sigmoid(y) - 0.5) * scale_x_y + 0.5 + grid_y) / nH
    bw = torch.exp(w) * anchor_w
    bh = torch.exp(h) * anchor_h
    bp = torch.sigmoid(p)
    preds = torch.stack([bx, by, bw, bh], -1)
    preds = torch.cat([preds, bp], -1)
    preds = preds.view(nB, -1, nattr)

    return preds

where pred is the output of the last convolutional layer right before a yolo layer, scale_x_y is 1 and stride is 32, 16 an 8.

it fails with error

RuntimeError: !node->kind().is_aten() && !node->kind().is_prim() && !node->kind().is_attr() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/serialization/export.cpp":453, please report a bug to PyTorch.

I don't understand the error. I can't see anything in that function that isn't either a pytorch tensor or a primitive. It's so close to working. Anybody have any ideas?? It would be great if a yolo model was fully ONNX exportable, without having to do much post processing with dynamic input size.

By the way, using that function works perfectly fine in normal execution, i.e. not running torch.onnx.export().

farleylai commented 3 years ago

@glenn-jocher

@Ezra-Yu yes that is correct. You are free to set it to False if that suits you better.

This seems to put the last layer (Detect) to work in the training mode for the ONNX export. Is there a special reason for that? Perhaps useful for other frameworks to import and train by default?

glenn-jocher commented 3 years ago

@farleylai you're free to modify export to suit your needs.

waheed0332 commented 3 years ago

@Ezra-Yu yes that is correct. You are free to set it to False if that suits you better.

But setting this to False throws error for coreml conversion. @dlawrences can you help me out regarding this?

waicool20 commented 3 years ago

If you are getting an error such as this during inference (device mismatch)

RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/models/yolo.py", line 47, in <fused code>
    _38 = (_5).forward(_37, )
    _39 = (_3).forward((_4).forward(_37, ), _30, )
    _40 = (_0).forward((_1).forward((_2).forward(_39, ), ), _38, _35, )
           ~~~~~~~~~~~ <--- HERE
    _41, _42, _43, _44, = _40
    return (_44, [_41, _42, _43])
  File "code/__torch__/models/yolo.py", line 73, in forward
    _52 = torch.sub(_51, CONSTANTS.c2, alpha=1)
    _53 = torch.to(CONSTANTS.c3, dtype=6, layout=0, device=torch.device("cpu"), pin_memory=False, non_blocking=False, copy=False, memory_format=None)
    _54 = torch.mul(torch.add(_52, _53, alpha=1), torch.select(CONSTANTS.c4, 0, 0))
                    ~~~~~~~~~ <--- HERE
    _55 = torch.slice(y, 4, 0, 2, 1)
    _56 = torch.expand(torch.view(_54, [3, 20, 20, 2]), [1, 3, 20, 20, 2], implicit=True)

Traceback of TorchScript, original code (most recent call last):
C:\Users\waicool20\Programming\python\yolov5\models\yolo.py(34): forward
C:\Python38\lib\site-packages\torch\nn\modules\module.py(534): _slow_forward
C:\Python38\lib\site-packages\torch\nn\modules\module.py(548): __call__
C:\Users\waicool20\Programming\python\yolov5\models\yolo.py(117): forward_once
C:\Users\waicool20\Programming\python\yolov5\models\yolo.py(97): forward
C:\Python38\lib\site-packages\torch\nn\modules\module.py(534): _slow_forward
C:\Python38\lib\site-packages\torch\nn\modules\module.py(548): __call__
C:\Python38\lib\site-packages\torch\jit\__init__.py(1027): trace_module
C:\Python38\lib\site-packages\torch\jit\__init__.py(873): trace
./models/export.py(35): <module>
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

You can work around it by manually editing the yolo.py file inside the exported torchscript archive

  def forward(self: __torch__.models.yolo.Detect,
    argument_1: Tensor,
    argument_2: Tensor,
    argument_3: Tensor) -> Tuple[Tensor, Tensor, Tensor, Tensor]:
    dev = argument_1.device                        <--- Add this line
    _45 = self.anchor_grid
    bs = ops.prim.NumToTensor(torch.size(argument_1, 0))

Then replace all the references of torch.device("cpu") to dev

Not sure if there's a better way to export it so it does something like this by default :/

zhiqwang commented 3 years ago

Hi @waicool20 , It seems that your model or input images are not on the same device when you are tracing, just make sure their device are same. It's not needed to edit the yolo.py the authors write here (I guess).

waicool20 commented 3 years ago

Hi @waicool20 , It seems that your model or input images are not on the same device when you are tracing, just make sure their device are same. It's not needed to edit the yolo.py the authors write here (I guess).

The model and input images are on the same devices, I have checked already through the debugger. But the exported model loads some constant tensors into cpu, leading to that error.

zhiqwang commented 3 years ago

The model and input images are on the same devices, I have checked already through the debugger. But the exported model loads some constant tensors into cpu, leading to that error.

Hi @waicool20 Oops, it's weird here.

waheed0332 commented 3 years ago

Converting Frontend ==> MIL Ops: 90%|████████████████████████████████████████▌ | 1051/1165 [00:00<00:00, 1457.09 ops/s] CoreML export failure:

Export complete (8.28s). Visualize with https://github.com/lutzroeder/netron.

When i set model.model[-1].export = False coreml export fails without and error. What could be the reason?

@zhiqwang @waicool20

zhiqwang commented 3 years ago

Hi @waheed0332 , Sorry, I did not have many experiences on CoreML.

chad-green commented 3 years ago

I've been trying to get a yolo model to be ONNX exportable with dynamic input size, while not having to manually add grid offsets or apply anchors. I have the following:
@torch.jit.script
def yolo_predict(pred, anchors, stride: int, scale_x_y: float):
    #transpose
    nA = anchors.size(0)
    nB, nC, nH, nW = pred.size()
    nattr = int(nC / nA)
    pred = pred.view(nB, nA, nattr, nH, nW).permute(0, 1, 3, 4, 2).contiguous()

    #make grid
    grid_x = torch.arange(nW).view(1, 1, 1, nW)
    grid_y = torch.arange(nH).view(1, 1, nH, 1)
    anchor_w = anchors[:, 0].view(1, nA, 1, 1).float() / (nW * stride)
    anchor_h = anchors[:, 1].view(1, nA, 1, 1).float() / (nH * stride)

    #apply grid
    x = pred[..., 0]
    y = pred[..., 1]
    w = pred[..., 2]
    h = pred[..., 3]
    p = pred[..., 4:]
    bx = ((torch.sigmoid(x) - 0.5) * scale_x_y + 0.5 + grid_x) / nW
    by = ((torch.sigmoid(y) - 0.5) * scale_x_y + 0.5 + grid_y) / nH
    bw = torch.exp(w) * anchor_w
    bh = torch.exp(h) * anchor_h
    bp = torch.sigmoid(p)
    preds = torch.stack([bx, by, bw, bh], -1)
    preds = torch.cat([preds, bp], -1)
    preds = preds.view(nB, -1, nattr)

    return preds
where pred is the output of the last convolutional layer right before a yolo layer, scale_x_y is 1 and stride is 32, 16 an 8.

it fails with error
RuntimeError: !node->kind().is_aten() && !node->kind().is_prim() && !node->kind().is_attr() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/serialization/export.cpp":453, please report a bug to PyTorch. 
I don't understand the error. I can't see anything in that function that isn't either a pytorch tensor or a primitive. It's so close to working. Anybody have any ideas?? It would be great if a yolo model was fully ONNX exportable, without having to do much post processing with dynamic input size.

By the way, using that function works perfectly fine in normal execution, i.e. not running torch.onnx.export().

@pfeatherstone Were you able to get it to work? Dynamic batching would be very helpful.

ultralytics / yolov5