LIVE object detection on custom model does not work well and has bugs

[HripsimeS]

Hello! @romovpa @kostmo @smacke @kit1980 @jeffxtang @IvanKobzarev @nairbv

I need your help to figure out how to improve LIVE object detection with my custom model. I trained my dataset with yolov5s model and pt model I converted to torchscript.ptl by doing the following modifications in https://github.com/ultralytics/yolov5/blob/master/export.py

fl = file.with_suffix('.torchscript.ptl') ts = torch.jit.trace(model, im, strict=False) d = {"shape": im.shape, "stride": int(max(model.stride)), "names": model.names} extra_files = {'config.txt': json.dumps(d)}
if optimize:
optimize_for_mobile(ts)._save_for_lite_interpreter(str(fl), _extra_files=extra_files) else: ts.save(str(fl), _extra_files=extra_files) return fl, None

I would like to mention that with my model the app was getting Lite Interpreter version error with PyTorch lib versions 1.10.0, therefore I modify build.gradle file by updating versions to 1.12.2 implementation 'org.pytorch:pytorch_android_lite:1.12.2' implementation 'org.pytorch:pytorch_android_torchvision_lite:1.12.2'

The app worked eventually with 1.12.2 versions, on pictures I get classes predictions although not too accurate. But LIVE detection is hopeless as you can see some detection's text on the top left side of the mobile screen but it does not really create a bounding box for the objects that are present in front of camera.

I downloaded and tested the app on your yolov5s.torchscript.ptl model you have in https://github.com/pytorch/android-demo-app/tree/master/ObjectDetection#1-prepare-the-model and it works fine with a good detection on pictures and LIVE camera.

So it is not clear for me the issue comes from my custom model or I still need to modify some settings in android object detection app to make it work accuratly on pictures and LIVE real time detection. If you need I even can share my model and classes files, so you can test on your side to see what is the issue. Look forward to hearing from you!

[kit1980]

Hi @HripsimeS. I'm not sure this repo is a good place to discuss problems with your custom model. I suggest to ask on https://discuss.pytorch.org/ or StackOverflow.

[NeighborhoodCoding]

I believe the custom model dose not work well in android, it is very nitch way to make custom model to run on real mobile device.

But finetunning the sample model will work better. When you try this although, I have some reference here https://pytorchvideo.org/docs/tutorial_accelerator_build_your_model

[HripsimeS]

@NeighborhoodCoding thanks for your advice! Is it really feasible to use those efficient blocks for mobile CPU in yolov5 training script? https://github.com/ultralytics/yolov5/blob/master/train.py Have you tried it before to use those efficient blocks for your model to boost the real time LIVE detection ?