search

lobe / iOS-bootstrap

Bootstrap your Lobe machine learning model with our iOS starter project.
https://lobe.ai
MIT License
98 stars 16 forks source link

Prediction on still images has worse performance on iOS than on Lobe desktop #22

Open technoplato opened 3 years ago

technoplato commented 3 years ago

Hello, love the app, first of all. This is how training should be!

I did run into a snag: I've successfully trained the model to recognize screenshots from audible, youtube, and apple podcasts and that model works very well on desktop using the "Play" section.

However, when I export the model and use it in the iOS Bootstrap, it does very bad at recognizing any screenshot accurately.

All I've done with code is replaced LobeModel with SavedModel directly from Lobe.

Any suggestions?

Video demo link: https://www.youtube.com/watch?v=_rtKnlmsxzM

technoplato commented 3 years ago

Disregard.

There is a popup when you go to export your model in the desktop application that asks if you'd like to optimize your model.

I clicked optimize the second go round and everything works flawlessly now. Thanks!

technoplato commented 3 years ago

Sorry Lobe people for the back and forth, I’m going to reopen this as the performance on device, even with the optimized model is much less accurate than that on the desktop application.

Any tips would be appreciated.

ellbosch commented 3 years ago

Thanks @technoplato—we will be deploying changes as soon as this week to fix these issues. You are correct to note that optimizing your model will not resolve this issue :)

technoplato commented 3 years ago

Well that's great to hear! Glad I'm not crazy.

As a side note, is Object Detection getting close or is that still pretty far out in the timeline?

Thanks!

ellbosch commented 3 years ago

Object detection is a rather large release and a top priority, but we can't say more than that. We are starting an insiders ring to test early features with object detection, if you are interested in joining email lobeai@microsoft.com.

technoplato commented 3 years ago

Totally understand. Quick question: are all models expected to have poor performance or just the CreateML?

On Tue, Feb 9, 2021 at 7:03 PM Elliot Boschwitz notifications@github.com wrote:

Object detection is a rather large release and a top priority, but we can't say more than that. We are starting an insiders ring to test early features with object detection, if you are interested in joining email lobeai@microsoft.com.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lobe/iOS-bootstrap/issues/22#issuecomment-776328739, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU2FGEXYFL5OVTILXYPW3LS6HELNANCNFSM4XJ4TGUQ .

ellbosch commented 3 years ago

For the particular bug you posted, it's a bug specific to image processing. There's actually nothing wrong with the Core ML model—it should behave as you would expect. We're just plugging in a bad input to the model which creates the weird results you're seeing.

technoplato commented 3 years ago

Now I’m a little more confused. Is the bug with the starter code then? Could you elaborate on what you mean about image processing?

On Wed, Feb 10, 2021 at 12:29 AM Elliot Boschwitz notifications@github.com wrote:

For the particular bug you posted, it's a bug specific to image processing. There's actually nothing wrong with the Core ML model—it should behave as you would expect. We're just plugging in a bad input to the model which creates the weird results you're seeing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lobe/iOS-bootstrap/issues/22#issuecomment-776453160, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU2FGE36AJEG4JKEES3E7TS6IKTRANCNFSM4XJ4TGUQ .

ellbosch commented 3 years ago

You are correct that it's a bug with the starter code. Here's a brief explanation on how iOS-bootstrap works:

  1. Video capture: first, the iOS device's camera transmits video feed.
  2. Image processing: a frame of the video feed needs formatting (with cropping and orientation adjustments) before we can predict.
  3. Prediction: the processed image is assigned a prediction from the Core ML model's classifier.

Step 2 above is what's broken, and should be fixed this week.

I highly recommend our docs if you'd like to learn more! Please let me know if you have any other questions.

technoplato commented 3 years ago

Haven’t had a chance to dig too deep into the code yet but just wanted to make sure that (2) is expected even when an image is selected from the image library.

On Wed, Feb 10, 2021 at 11:22 AM Elliot Boschwitz notifications@github.com wrote:

You are correct that it's a bug with the starter code. Here's a brief explanation on how iOS-bootstrap works:

  1. Video capture: first, the iOS device's camera transmits video feed.
  2. Image processing: a frame of the video feed needs formatting (with cropping and orientation adjustments) before we can predict.
  3. Prediction: the processed image is assigned a prediction from the Core ML model's classifier.

Step 2 above is what's broken, and should be fixed this week.

I highly recommend our docs https://docs.lobe.ai/docs/improving/improving if you'd like to learn more! Please let me know if you have any other questions.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lobe/iOS-bootstrap/issues/22#issuecomment-776830740, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU2FGAHP27FQ7ROYWUS37TS6KXFBANCNFSM4XJ4TGUQ .

ellbosch commented 3 years ago

Good question! Yes, we also format images selected from device library before sending to prediction (i.e. we need images formatted into a square aspect ratio).

technoplato commented 3 years ago

Perfect. So images chosen from the library will suffer the same bug. Will be looking forward to the update patiently. Thank you guys for your support here and for creating such an accessible and seamless tool for such a powerful technology!

On Thu, Feb 11, 2021 at 12:43 PM Elliot Boschwitz notifications@github.com wrote:

Good question! Yes, we also format images selected from device library before sending to prediction (i.e. we need images formatted into a square aspect ratio).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lobe/iOS-bootstrap/issues/22#issuecomment-777670178, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU2FGF5OFSQC7DJKKHLDCDS6QJNHANCNFSM4XJ4TGUQ .

ellbosch commented 3 years ago

Your welcome :) the PR is here if you'd like a sneak peek of the fix: #23

technoplato commented 3 years ago

Super clean PR - love the refactoring and more information.

Unfortunately, it actually looks like the same issue of inaccurate detection (of images that work on Desktop) as develop :(

Can I help provide any information for validating / resolving the issue?

ellbosch commented 3 years ago

Shoot 😩 I'm sorry this PR didn't fix things. My guess is I was focusing too much on fixing image processing bugs for the camera feed rather than image preview mode, although I know both modes are more reliable with this upcoming PR.

I'll try to repro your specific project. I think I know why this might still be breaking on your end. I'll reach back out to you via this thread when I have an update here.

technoplato commented 3 years ago

Please don’t apologize!! It’s an awesome PR and I’m getting everything for free!

I made a gif replicating the issue. I can also share my mode if it helps just ping me :)

On Thu, Feb 11, 2021 at 8:05 PM Elliot Boschwitz notifications@github.com wrote:

Shoot 😩 I'm sorry this PR didn't fix things. My guess is I was focusing too much on fixing image processing bugs for the camera feed rather than image preview mode, although I know both modes are more reliable with this upcoming PR.

I'll try to repro your specific project. I think I know why this might still be breaking on your end. I'll reach back out to you via this thread when I have an update here.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lobe/iOS-bootstrap/issues/22#issuecomment-777899544, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU2FGE4GDUZKB3VATWQVGLS6R5FDANCNFSM4XJ4TGUQ .

ellbosch commented 3 years ago

Hello @technoplato! I believe PR #25 might have resolved this bug. Would you mind validating this by testing the u/elbosc/63/model-preprocessing branch with your project?

technoplato commented 3 years ago

I can’t wait. I saw release .9 and am going to try and rerun my mode tomorrow :)

On Wed, Mar 24, 2021 at 10:49 PM Elliot Boschwitz @.***> wrote:

Hello @technoplato https://github.com/technoplato! I believe PR #25 https://github.com/lobe/iOS-bootstrap/pull/25 might have resolved this bug. Would you mind validating this by testing the u/elbosc/63/model-preprocessing branch with your project?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lobe/iOS-bootstrap/issues/22#issuecomment-806323288, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU2FGAQANCXXZHDS4UZABDTFKQENANCNFSM4XJ4TGUQ .

technoplato commented 3 years ago

Hey @ellbosch I'm running into 'Network Disconnected' errors when trying to download the application either through https://lobeprod.azureedge.net/downloads/macos/Lobe.zip or through the home page (which I assume uses the same endpoint)

Can you send me the zipped application directly at halfjew22@gmail.com? Really looking forward to playing with the new tool and demos

EDIT: I'm trying to download it through my phone and it's working so far. Strange.

mbeissinger commented 3 years ago

@technoplato sent you an email with a link as well

technoplato commented 3 years ago

Unfortunately, no dice here either. I'm going to upload my training set and see if I'm just doing something wrong: https://drive.google.com/drive/folders/1nGTCSE9dsgz1ZJDVgc4rMGmBoyNJ-wmk?usp=sharing

I'd upload the model as well but I'm suspicious that's what I'm messing up.

Received the email @mbeissinger. Going to try exporting a new model, but after trying a v.8 model with the new iOS bootstrap that @ellbosch linked above, I'm still getting poor results

mbeissinger commented 3 years ago

@technoplato is the model in iOS giving different results than from Lobe's Use tab?

One thing from quickly looking at the dataset -- it is probably overfitting since most of the images in each class look very similar to each other. I would try to prune out images that pretty much look the same and try to capture more varied examples. Here is our docs section about improving the model/dataset: https://docs.lobe.ai/docs/improving/improving#why-is-lobe-not-predicting-well-on-new-images-in-use

technoplato commented 3 years ago

@mbeissinger Thank you for the suggestion.

I've got 99% accuracy within the Lobe Desktop application. The issue arises when I attempt to use the model in the ios-bootstrap. I believe Lobe Desktop's almost perfect accuracy implies that, despite a relatively repetitive dataset, the model "functions", even if just in the Desktop application

mbeissinger commented 3 years ago

@technoplato ah ok! So is the CoreML model not loading/giving any results or are they different results than in Lobe?

technoplato commented 3 years ago

Correct. The CoreML model is giving results but they’re all incorrect and different than Lobe.

On Fri, Mar 26, 2021 at 3:18 PM Markus Beissinger @.***> wrote:

@technoplato https://github.com/technoplato ah ok! So is the CoreML model not loading/giving any results or are they different results than in Lobe?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lobe/iOS-bootstrap/issues/22#issuecomment-808456894, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU2FGC5SUH2KXAE47PGOSDTFTMW7ANCNFSM4XJ4TGUQ .

mbeissinger commented 3 years ago

@technoplato ok great I'll see if we can replicate

technoplato commented 3 years ago

Awesome let me know! I’ll be available to try other things as well to help troubleshoot if you’d like / need.

Hopefully operator error here. I’m driving Elliott nuts 😜

On Fri, Mar 26, 2021 at 3:20 PM Markus Beissinger @.***> wrote:

@technoplato https://github.com/technoplato ok great I'll see if we can replicate

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lobe/iOS-bootstrap/issues/22#issuecomment-808458146, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU2FGELR2VAWISRFBXXCL3TFTNBDANCNFSM4XJ4TGUQ .

ellbosch commented 3 years ago

@technoplato the only thing driving me nuts is that this bug still exists. Sorry we still haven't resolved this!

technoplato commented 3 years ago

We’re you guys able to verify on your end?

I tried to run the react app but didn’t have any luck there. I wonder if the node example would work? Just to eliminate the possibility of operator error.

On Tue, Mar 30, 2021 at 1:16 PM Elliot Boschwitz @.***> wrote:

@technoplato https://github.com/technoplato the only thing driving me nuts is that this bug still exists. Sorry we still haven't resolved this!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lobe/iOS-bootstrap/issues/22#issuecomment-810434637, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU2FGCJ4BEVDLDL7YH5WS3TGIBNHANCNFSM4XJ4TGUQ .

mbeissinger commented 3 years ago

@technoplato @ellbosch I verified we have automatic tests for the model outputs going through the converters, so likely a bug in the ios processing. There will be some slight differences in outputs due to the different frameworks but they should not be major. I'm working on adding a file picker to the web-bootstrap so it isn't just the webcam -- you should be able to test the same image within Lobe and in the starter project soon. Can you also try it with the Lobe Connect local API using the same image as something you drag into the Use tab?

technoplato commented 3 years ago

Sorry for my delay @mbeissinger finally got around to this and the Lobe Connect Run mode works perfectly!

python3 lobeconnect.py
correct label: youtube.png
predicted label:    youtube
confidence:     1
correct label: audible.png
predicted label:    audible
confidence:     1

Confirming the React App works as well. Thanks for the updates to allow static image selection!