iDetection v7.7 Release

[glenn-jocher]

🚀 Feature

We've released iDetection v7.7 on the iOS app store (https://apps.apple.com/app/id1452689527)! This release features all four v3.0 models YOLv5s/m/l/x running with hardswish activations for the first time. This update also brings some supercool features we've been working on:

• Modify Confidence Threshold and NMS IoU Threshold in realtime user sliders and instantly see the effects:

  

• Rotation capability. device orientation is now passed in realtime to the inference handler, so the app is more robust to landscape and upside down inference than before. Vertical and upside down inference works best, as we export our models for vertical rectangular inference, so when used in landscape mode the app is resizing the horizontal video into the vertical inference, which is not ideal but still produces results.

  

[NanoCode012]

Hi @glenn-jocher , I tested on my 11pro. Very nice fps (19-25 FPS) and accuracy (90+ on recognizable objects).

It does get quite hot in a matter of minutes though (especially when changing the thresholds). I saw a quick battery dip. Another thing I’ve noticed is that the labels of some bounding boxes goes out of the phone’s view (maybe it could be clamped in).

[glenn-jocher]

@NanoCode012 yes, you've noticed the effect of thermal throttling. This is a really interesting topic. I'm not a hardware expert, but this is my understanding:

iOS devices are passively cooled, which means they use the mass of the device to absorb thermal heat away from the CPU, and then convection (and to a lesser degree infrared radiation) from the heated device out to the atmosphere to pass this heat onwards into the surrounding environment. The iPhone is thus incapable of responding to increased workloads with increased active cooling (fan RPMs), the way a GPU or desktop computer can.

Apple surely has smart thermal management, battery management, and throttling software onboard to regulate the CPU, GPU and ANE throughputs in order to maintain the device within thermal constraints. This means that the device will enjoy it's full computational capacity only for short time periods before it begins to overheat, at which point the throttling logic will reduce the available FLOPS low enough to ensure a steady state device temperature below threshold.

In practice the longest I've run iDetection has been 1 hour, on an iPhone 11, cover removed, not in sunlight, 25C day with no wind, starting from a 100% state of battery charge. My observation was that full speed inference (30 FPS) was available for the first 10 minutes, after which FPS reduced to about 11 FPS and maintained this steady state performance for an additional 50 minutes, until I ended the experiment. Battery usage over the 60 minutes was 40%, so 60% charge remained.

In 60 minutes the iPhone was able to process about 40,000 images, and captured about half a million instances of people walking along a street (about 10 people per frame).

[glenn-jocher]

@NanoCode012 also yes, you've noticed the different image and video source sizes. The video input is 4k, or 16:9 aspect ratio, whereas the screen display is actually narrower, so there are video regions on the left and right side that are not visible on screen.

The alternative to view the full video would be to have black bars at the top and bottom (like the native camera app actually), but I decided against this as the full screen video seemed to provide a better user experience.

[glenn-jocher]

iDetection v7.8 preview screen

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlcool]

Where is the source code?

[fanatic-studio]

Where is the source code?

[glenn-jocher]

@jlcool @view-team we have not open-sourced the iDetection source code, however soon you'll be able to preview your own custom models in iOS and Android using the Ultralytics HUB. Please visit https://ultralytics.com/hub to learn more.

[yusufoos]

@glenn-jocher as per "Vertical and upside down inference works best, as we export our models for vertical rectangular inference" - How can we export the model for horizontal inference instead so that landscape works better? Looking for support or guidance on this since Landscape/Horizontal is very important for my use-case.

Thanks!

[glenn-jocher]

@yusufoos yes, you can export any shape you want here with the export.py --imgsz argument, i.e. --imgsz 640 640: https://github.com/ultralytics/yolov5/blob/436ffc417ac2312de18287ddc4f87bdc2f7f5734/export.py#L479

FYI our height-width order is backwards compared to normal conventions, so these will be reversed in a fix PR sometime soon.

I think for Android for example the native stream shape is landscape at a different aspect ratio, maybe 256x320 landscape there works best.