Convert the model to tensorflow

[TTMRonald]

Hi, I want to inference in tensorflow. Did you try converting both model and parameters to tensorflow? Does the ROIAlign layer support onnx? Thanks.

[tianzhi0549]

@TTMRonald We did not try to convert it to tensorflow. However, it should be very easy because FCOS is only involved with standard conv layers. I suppose that conv layers should be well-supported by ONNX. FCOS does not need ROIAlign since it is one-stage.

[TTMRonald]

@tianzhi0549 Thank you for your explanation. I will try converting model to tensorflow.

[TTMRonald]

@tianzhi0549 Hi, I meet an error, when i tried to convert fcos model to onnx. My pytorch is nightly 1.1.0.

RuntimeError: Only tuples, lists and Variables supported as JIT inputs, but got BoxList

Do you have any suggestions?

[tianzhi0549]

@TTMRonald Can you post the full calling stack here? BoxList is a wrapper of bounding-boxes, defined at https://github.com/tianzhi0549/FCOS/blob/fdfc912474e2fff863b8d2dadbba11567fafd758/maskrcnn_benchmark/structures/bounding_box.py#L9.

[TTMRonald]

@tianzhi0549 Hi, I add the convert code after the "predictions = self.model(image_list)".

https://github.com/tianzhi0549/FCOS/blob/fdfc912474e2fff863b8d2dadbba11567fafd758/demo/predictor.py#L205

The convert code is dummy_input = torch.autograd.Variable(torch.randn(1, 3, 640, 800).cuda()) torch.onnx.export(self.model, dummy_input, "convert_models/onnx_model.onnx")

The code have an error, when the code runing in the "torch.onnx.export()" The error is

RuntimeError: Only tuples, lists and Variables supported as JIT inputs, but got BoxList.

Is there any way to solve the compatibility problem between the the model output type (BoxList) and type torch.onnx.expor requested? Thanks.

[tianzhi0549]

@TTMRonald You can't directly apply export to self.model. self.model includes operations such as NMS that cannot be automatically converted. I think you should only convert the CNN layers in the model.

[TTMRonald]

@tianzhi0549 I tracked this code and found that self.model was defined at https://github.com/tianzhi0549/FCOS/blob/fdfc912474e2fff863b8d2dadbba11567fafd758/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py#L16

I didn't find NMS. Can you tell me where torch.onnx.export should be added? Thanks.

[tianzhi0549]

@TTMRonald NMS happens after https://github.com/tianzhi0549/FCOS/blob/fdfc912474e2fff863b8d2dadbba11567fafd758/maskrcnn_benchmark/modeling/rpn/fcos/fcos.py#L127.

Therefore, I think you should only convert the model before this line (inclusion) to ONNX.

[TTMRonald]

@tianzhi0549 I found self.head to contain convolution operations. The final conversion model should be two modules containing self.backbone and self.rpn (not included NMS). The final conversion model output contains box_cls, box_regression, centerness, locations (used for NMS). The conversion is more troublesome, Can you provide a demo to convert onnx? Thanks.

[tianzhi0549]

@TTMRonald You've got the point. I am not familiar with ONNX and can't give you the specific code for now.

[TTMRonald]

@tianzhi0549 Thank you for your help.

[TTMRonald]

Update! I tried several versions of the pytorch and there were different errors. I add the model output (return [box_cls, box_regression, centerness, locations], None) after this line. https://github.com/tianzhi0549/FCOS/blob/fdfc912474e2fff863b8d2dadbba11567fafd758/maskrcnn_benchmark/modeling/rpn/fcos/fcos.py#L128 I add the convert code after this line. https://github.com/tianzhi0549/FCOS/blob/fdfc912474e2fff863b8d2dadbba11567fafd758/demo/predictor.py#L205 The convert code is dummy_input = torch.autograd.Variable(torch.randn(1, 3, 640, 800).cuda()) torch.onnx.export(self.model, dummy_input, "convert_models/onnx_model.onnx") But the convert code meets an error. In the PyTorch 1.0.0 from a nightly.

ValueError: Auto nesting doesn't know how to process an input object of type maskrcnn_benchmark.structures.image_list.ImageList. Accepted types: Tensors, or lists/tuples of them

In the PyTorch 1.1.0 from a nightly.

RuntimeError: Failed to export an ONNX attribute, since it's not constant, please try to make things (e.g., kernel size) static if possible

In the PyTorch 1.2.0 from a nightly.

KeyError: 'rsqrt'

In the PyTorch 1.1.0. The error was resolved in the PyTorch 1.0.1. But FCOS/INSTALL.md describes that maskrcnn-benchmark will not work with 1.0 nor 1.0.1.

RuntimeError: Failed to export an ONNX attribute, since it's not constant, please try to make things (e.g., kernel size) static if possible

Can anyone give me some advice? Thanks.

[tianzhi0549]

@TTMRonald Please ignore the warnings about pytorch versions. I suggest you can first make the input to self.model(image_list) as a pytorch tensor. The ImageList cannot be recognized by ONNX.

[TTMRonald]

@tianzhi0549 I . The input I set for the self.model is a Variable, when I try convert to onnx. dummy_input = torch.autograd.Variable(torch.randn(1, 3, 640, 800).cuda())

Bu I suspect that the ImageList is defined inside this self.model. https://github.com/tianzhi0549/FCOS/blob/c1eee2c3990319b7076e79332854c808cb7191f8/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py#L48

[tianzhi0549]

@TTMRonald Yes. You should modify the code related to ImageList.

[TTMRonald]

Update！ Fixed an issue with ImageList, but there were some new issues that needed to be modified to define torch.onnx.symbolic.

/home/.local/lib/python3.6/site-packages/torch/tensor.py:427: RuntimeWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results). 'incorrect results).', category=RuntimeWarning) /home/.local/lib/python3.6/site-packages/torch/onnx/utils.py:501: UserWarning: ONNX export failed on ATen operator rsqrt because torch.onnx.symbolic.rsqrt does not exist .format(op_name, op_name)) /home/.local/lib/python3.6/site-packages/torch/onnx/utils.py:501: UserWarning: ONNX export failed on ATen operator reshape because torch.onnx.symbolic.reshape does not exist .format(op_name, op_name)) /home/.local/lib/python3.6/site-packages/torch/onnx/utils.py:501: UserWarning: ONNX export failed on ATen operator group_norm because torch.onnx.symbolic.group_norm does not exist .format(op_name, op_name)) Traceback (most recent call last): File "demo/fcos_demo.py", line 110, in main() File "demo/fcos_demo.py", line 101, in main composite = coco_demo.run_on_opencv_image(img) File "/opt/detection/fcos/demo/predictor.py", line 175, in run_on_opencv_image predictions = self.compute_prediction(image) File "/opt/detection/fcos/demo/predictor.py", line 216, in compute_prediction torch.onnx.export(self.model, dummy_input, "conver_models/FCOS_R_101_FPN_2x.onnx") File "/home/.local/lib/python3.6/site-packages/torch/onnx/init.py", line 27, in export return utils.export(*args, **kwargs) File "/home/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 104, in export operator_export_type=operator_export_type) File "/home/.local/lib/python3.6/site-packages/torch/onnx/utils.py", line 287, in _export proto, export_map = graph.export(params, _onnx_opset_version, defer_weight_export, operator_export_type) RuntimeError: ONNX export failed: Couldn't export operator aten::rsqrt

[tianzhi0549]

@TTMRonald It seems that rsqrt is not implemented by ONNX. Please replace it with the reciprocal of sqrt. rsqrt is used at https://github.com/tianzhi0549/FCOS/blob/c1eee2c3990319b7076e79332854c808cb7191f8/maskrcnn_benchmark/layers/batch_norm.py#L20.

[TTMRonald]

@tianzhi0549 Thanks. Fixed the rsqrt error. It seems that there are fewer operators implemented by ONNX. Also need to define reshape and group_norm.

[tianzhi0549]

@TTMRonald OK. Just define them by yourself.

[TTMRonald]

@tianzhi0549 Yes. Also ask if FCOS requires segmentation annotation to generate mask annotations, when train your own datasets.

[tianzhi0549]

@TTMRonald No. FCOS only uses bounding boxes annotations.

[TTMRonald]

@tianzhi0549 OK. Thanks.

[drcege]

@TTMRonald Just an aside, why are you still using Variable? From pytorch 0.4, torch.Tensor and torch.autograd.Variable have merged. (PyTorch 0.4.0 Migration Guide)

[TTMRonald]

@drcege Thank you for your reminder, I used it according to the ONNX official example.

[LCWdmlearning]

hello，Can you share the code and details for generating onnx? I got the error is :

ValueError: Auto nesting doesn't know how to process an input object of type maskrcnn_benchmark.structures.image_list.ImageList. Accepted types: Tensors, or lists/tuples of them

[tianzhi0549]

@LCWdmlearning Please refer to https://github.com/tianzhi0549/FCOS/issues/57#issuecomment-499757720.

[xscjun]

@tianzhi0549 I found self.head to contain convolution operations. The final conversion model should be two modules containing self.backbone and self.rpn (not included NMS). The final conversion model output contains box_cls, box_regression, centerness, locations (used for NMS). The conversion is more troublesome, Can you provide a demo to convert onnx? Thanks.

@tianzhi0549 I found self.head to contain convolution operations. The final conversion model should be two modules containing self.backbone and self.rpn (not included NMS). The final conversion model output contains box_cls, box_regression, centerness, locations (used for NMS). The conversion is more troublesome, Can you provide a demo to convert onnx? Thanks.

Hi, I want know how many models you finally convert, two ? (self.backbone and self.rpn without nms ),can all the layers including convolution be converted to one single model ?

[raviv]

@TTMRonald Did you manage to export to ONNX? If yes, can you share your changes?

[yhpwsid]

can you share your changes for converting the model?thanks @TTMRonald