Android app rework using Kotlin and more modern approach

[paschalis-mpeis]

There must be something wrong with the configuration of the camera activity.

[Gunock]

There must be something wrong with the configuration of the camera activity.

Could you provide more details (logs, screenshots, etc.)?

[paschalis-mpeis]

Sure, here you are:


Do you want anything more specific?

[Gunock]

As for now I'll try to replicate this on one of my devices.

[Gunock]

Ok, it's not happening on my devices. Could you provide some informations about the device you're using? And were you holding the device horizontally or vertically? (the app is locked in single position so I can't tell from the screenshot)

[paschalis-mpeis]

Hmm.. I tested this on another device (Samsung S20 I think). On that one it works fine!

But on my other device (CAT S62) it works just like the screenshot I've attached.

More stats: CAT S62:

• Frame: 864x480
• Crop: 416x416

Samsung S20:

• 1280 x 720
• Crop: 416, 416

[Gunock]

Ok, the CAT is not using 16:9 for camera images. Maybe I can force similar resolution on my phone.

[paschalis-mpeis]

On the original java code (not the Kotlin MVVM) I get the below:

S62: (the device that had issue

• Frame 640 x 480

Samsung S20:

• Frame: 1280 x 720

And it works well on both devices. So my guess is that the image gets distorted on the CAT S62.

[Gunock]

You could put a breakpoint in YoloV4Detector class and view the scaledBitmap variable in android studio. Colors will be shifted but otherwise the image should look normal. If not then I guess something is not right with the Image to Bitmap conversion.

I attach a screenshot where to put the breakpoint

[paschalis-mpeis]

I'll try it now.

BTW was this a porting from hunglc007 or a fresh implementation based on it? Because that one is using a different frame size and works OK.

[Gunock]

It's mostly fresh implementation with parts from hunglc007. The old implementation was a pain in the ass to convert to Kotlin and it used archaic camera APIs.

[paschalis-mpeis]

Samsung S20: Looks fine

CAT S62 its just garbage: bitmap:

scaled bitmap:

[Gunock]

Ok, that got to be conversion problem. The androidx camera provides images in YUV format and there is a code which converts it to RGB. Something must be wrong with the YUV buffer (I had these problems previously). I'll take a look a it later. The problem is most probably in ImageToBitmapConverter class, imageToBitmap method.

[paschalis-mpeis]

Could it be that the frame size (864x480) is messing around with the YUV?

[Gunock]

Most probably yes. Usually it has something to do with the buffer size being some funny number.

[paschalis-mpeis]

Forcing the rationto AspectRatio.RATIO_4_3 in DetectorActivity seems to be working. Thank you so much for the help so far.

BTW the below distortion between the bitmap and the scaledBitmap is intended?

[Gunock]

The image is resized without maintaining aspect ratio. The same thing was done in the old implementation. I could replace it with padding instead but I don't know if it is an idea worth pursuing.

[Gunock]

I didn't factor the stride in image conversion, thats why for some resolution images are corrupted. I'm working on a fix

[Gunock]

@Paschalis Check my fix. There should be no problems for 16:9 on your CAT phone. Still 4:3 is usually native format for cameras so stick to that for later use. Please let me know if the fix works on your devices.

[paschalis-mpeis]

Thanks @Gunock for the update! It does work, I can confirm that:

BTW, as a follow-up to my previous comment, yolo accepts the resized image without aspect ratio? That is the bottom image? As it needs square images?

If so, and given that training happens on unstretched images, do you believe that a left/right white padding would improve detection? I'm asking as I'm thinking of implementing this feature if it improves inference.

[Gunock]

You can either train yolo on non square images or add padding. I guess padding would improve inference but that's just guessing.

EDIT: Maybe right/bottom padding would be better. Result position transposing should be easier and adding padding also should be easier (put image in position 0,0).

[paschalis-mpeis]

Thanks for the tip! But then, with right/bottom padding wouldn't the getDetections return Detections on offset positions?

Or is nms supposed to be "normalizing" Detections again? (because, the current approach could make the Detection rectangles wider).

[Gunock]

NMS stands for Non-maximum Suppression. It's a way of removing overlapping boxes.

YOLO operates on normalized coords (i.e. positions and dimensions are expressed in percent's). There's some code in getDetections that translates YOLO bounding box to rectangle (https://developer.android.com/reference/android/graphics/RectF).

Whatever you do with padding there will be needed some offset correction in bounding boxes.

[paschalis-mpeis]

Hi @Gunock ,

It's me again! Unfortunately with a custom model I get zero detections on your MVVM fork (commit @1ecc22c), whereas the original (non-mvvm) work is OK.

In particular, I tested 3 models:

1. coco (has 80 classes): works on both versions
2. custom model (has 29 classes): works on both versions
3. custom model (has 99 classes): works only on the non-mvvm version

In contrast with the last time, the above issue persists on both devices (i.e., it does not work on both a CAT S62 and a Samsung S20)

Can you guide me or give me some hints (once again) on how to resolve this? Also, it seems I can't open an issue on your repository.

Cheers, Paschalis

[Gunock]

Please contact me by mail ( asz.czyk.dev@gmail.com ). I will need more information.

[Gunock]

@Paschalis I have migrated from YUV to ARGB in CameraX's ImageAnalysis (latest version added support for it). This should resolve any issues with image color format conversion. I have also added image scaling that retains aspect ratio. This resolved issues with zero detections when using your custom model.