image tracking works at very random times on iOS...

[ffd8]

Amaazing lib and wild what it can do! Nevertheless, slowly going a little crazy, as I can't figure out why this works perfectly on my desktop (MBP, any browser), but on iOS (iphone 13 and testing older) – it only works every random nth time... Sometimes it works 4-5 times in a row, then it can equally do nothing and be stuck on scanning the same number of times or more (closing tab, reopening doesn't solve it). I've tried loading scripts in a waterfall but that didn't solve it (being just over 5mb with my combination of p5.js + hydra). Saw an issue about 'target confidence' which would be really interesting and potentially helpful to track when it works and fails.

My issue can be tested out in debug mode here, using the following image printed out for tracking.

Just loaded the latest 1.1.5 with the hope my issue would be solved, but no luck... Any experience of this happening? Any tips to debug further once 'AR Ready' is done and it's scanning? Or techniques to re-init the right components if nothing is tracked every x seconds? My only guess is that maaybe the target.mind file isn't always ready on time? (not sure if this can be preloaded) or where else I should look? Thanks for any tips!

[hiukim]

can do a screen recording to show the error?

[ffd8]

here's a short recording that captures my experience... works 2-4 times.. then not for a handful of times.. then again.. somehow very random (just refreshing the page, so i doubt i'd run into any loading issues since libs are likely cached...):

https://user-images.githubusercontent.com/570957/183502419-c54d248e-017e-4b03-8090-b05976bce556.mp4

[hiukim]

Is it possible it's still loading up and detecting? you refreshed the page pretty fast

[ffd8]

Was refreshing quickly just to try and pack a few tries into a short recording.. but always waited for the arReady event listener. same result if waiting - it seems to either recognize it right away or not at all... tried playing with the warmupTolerance to 1 or 2.. not quite sure if I see a difference. The targets.mind file is 750kb (maybe much bigger than it should be?), as the image I uploaded is 2480 × 3508px. That's probably total overkill? Guess I should test with a lower res image too incase it helps it get more constant positives? Or something else to tweak for a more consistent recognition?

[ffd8]

Just recompiled a .mind file from an image that's 496 × 702 (poster format), which created a 370kb .mind file. I want to say it's working a little better, but still not recognized 30-50% of the time... another idea, was if there's an easy 're-init' function to call after an interval check? If the page is being loaded, then the person is trying to see the AR experience.. once everything is loaded, if nothing is seen within x seconds, it would be good to try and refresh/init the search for a target now that everything is definitely loaded. I tried calling a .pause(true) .... .start() but didn't seem to make a difference.

[ffd8]

Aha! I think I found the issue/bug?! One of the times it didn't recognize, I accidentally rotated my phone enough to change it from portrait to landscape mode and then it worked! Rotating back to portrait, didn't work, back to landscape, worked!? Each time I reload and it doesn't load in portrait, it works fine when rotating 90deg.... I wonder if either iOS/Safari camera is at fault or what orientation it's sending the website, or if it's something within the lib?? Makes me wonder if it's very costly to also compare targets at 90deg in parallel just to find the thing??

edit: argh, but screen.orientation isn't available on the device.. since it requires asking permission for device orientation. it can however be determined from the width/height of screen. Not sure if MindAR is using orientation or dimensions?? if this is where the issue could be solved?

[hiukim]

If you rotate the phone after it's being initialized, it won't work. But that doesn't seem to be the problem according to what you described.

[ffd8]

Indeed – I'm not having the issue that it stops working, but rather that it very often never starts to work in the given orientation (ie holding the phone in portrait mode), however turning the device then starts to work = it was somehow setup for landscape mode, even though I was in portrait mode. This is now the 'answer' 100% of the time that the AR is not loading. So I can hold the phone in portrait mode, if it doesn't work, switch to landscape where it works, switch back to portrait - doesn't work, back to landscape where it continues working. Very strange. I wonder if the library needs a good init option where one can manually tell it (or override) the orientation based on a comparison of width and height dimensions (even though an automated check within lib should work). Another option, is it feasible to compare for pixels in both orientations of the image to webcam comparison? That way it would work in either all the time?

I'll see if iOS lets me record a video that changes orientation like this, but it's the same as described above.

[ffd8]

@hiukim here's an update - works EVERY time when the phone is loaded in landscape orientation, which matches why it also worked every time on my laptop. Since this is about an animated poster.. could you point out where I could force it to always use portrait rather than landscape? Or better yet a way I could set which fixed orientation it has upon loading (after making my own client screen comparison).

I wonder if the portrait iOS video issue is related to this SO and such a config option could also solve issue's #212 + #238 ?

[hiukim]

@ffd8 I'm not sure why this is happening to you, and I couldn't replicate the issue. so I'm not sure how to fix that at the moment. I might have some idea but I need try out.

The issue #238 is a different (and known) problem. The library cannot handle orientation changes after initiation.

The issue you are having seems to be having a webcam stream at landscape mode even though the device is portrait.

I wonder does it happen to only one of your devices? what device are you using and what iOS version? Perhaps you have other iOS devices to try out as well?

[ffd8]

The issue you are having seems to be having a webcam stream at landscape mode even though the device is portrait.

This sounds like what might be happening and I'm now wondering if this could be coming from a-frame or threejs or this lib?? It's not quite clear to me, who/which is initializing the webcam, although I do see it within this mindar lib source (just wondering if any other flags on the html5 video would help). It's also a problem on a friends iOS (don't have exact specs for that, but I believe iPhone 11 and latest iOS). As for myself, using an iPhone 13pro 15.3 (updating to 15.6 now to see if that's the issue). I'll try to test some other iPhones of friends tomorrow...

Within these issues, stumbled across mentions of 'zappar' the non-open-source tool, which also has an a-frame example. There it works fine in both portrait + landscape, which makes me wonder if they've potentially fixed something ios related for a-frame that could be the issue?? Would def prefer to use mind-ar (open source, for teaching etc). Also just updated to a-frame 1.3.0, but sadly didn't fix it.

[hiukim]

I guess it's probably not related to AFRAME. Just to confirm, you also have this problem when running the examples given in the doc, right?

e.g. https://hiukim.github.io/mind-ar-js-doc/more-examples/threejs-image or https://hiukim.github.io/mind-ar-js-doc/examples/minimal

[ffd8]

Eeeek - sorry for dragging this issue on and on.. just tested your examples and they are working just fine in portrait mode.. then tested a step further, using p5.js as I am, which worked fine... ahhh = perhaps there's a variable conflict within my own code and the minified libraries for a-frame, threejs, mind-ar... time to dig through with a comb and/or rebuild step by step to find the conflict. Since it's obviously now on my end, will close the issue.

Thanks again for ping-ponging with me to try and solve it.

[hiukim]

no problem. Glad we found the issue. Do share you findings if you figure out the root cause. I would be interested to know.