search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
26.95k stars 5.1k forks source link

Mediapipe CodePens don't run on iOS Safari #1427

Closed ionif closed 3 years ago

ionif commented 3 years ago

Hello all,

I have a project using Mediapipe Hands on iOS and I've been trying to update from the tfjs model to the new Mediapipe api but even when I enable WebGL2, it still fails to work. I've made sure I'm asking permission using navigator.getmedia properly. Wondering if anyone has any ideas on what's going wrong.

Here's the codepen that I'm testing: https://codepen.io/aionkov/pen/MWjEqWa

Here's the console:

[Warning] I1223 11:05:16.032000 1 gl_context_webgl.cc:146] Successfully created a WebGL context with major version 3 and handle 3 (hands_solution_wasm_bin.js, line 9) [Warning] I1223 11:05:16.034000 1 gl_context.cc:340] GL version: 3.0 (OpenGL ES 3.0 (WebGL 2.0)) (hands_solution_wasm_bin.js, line 9) [Warning] W1223 11:05:16.034000 1 gl_context.cc:794] Drishti OpenGL error checking is disabled (hands_solution_wasm_bin.js, line 9) [Warning] E1223 11:05:16.711000 1 calculator_graph.cc:775] INTERNAL: CalculatorGraph::Run() failed in Run: (hands_solution_wasm_bin.js, line 9) [Warning] Calculator::Open() for node "handlandmarktrackinggpuhandlandmarkgpuInferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\x84\x02\x08\r\x12\xff\x01\x43\x61lculator::Open() for node \"handlandmarktrackinggpuhandlandmarkgpuInferenceCalculator\" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50'] (hands_solution_wasm_bin.js, line 9) [Warning] F1223 11:05:16.712000 1 solutionswasm.embind.cc:585] Check failed: ::util::OkStatus() == (graph->WaitUntilIdle()) (OK vs. INTERNAL: CalculatorGraph::Run() failed in Run: (hands_solution_wasm_bin.js, line 9) [Warning] Calculator::Open() for node "handlandmarktrackinggpuhandlandmarkgpuInferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\x84\x02\x08\r\x12\xff\x01\x43\x61lculator::Open() for node \"handlandmarktrackinggpuhandlandmarkgpuInferenceCalculator\" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50']) (hands_solution_wasm_bin.js, line 9) [Warning] Check failure stack trace: (hands_solution_wasm_bin.js, line 9) [Warning] undefined (hands_solution_wasm_bin.js, line 9) [Error] Unhandled Promise Rejection: RuntimeError: abort(undefined) at jsStackTrace@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands_solution_wasm_bin.js:9:67558 stackTrace@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands_solution_wasm_bin.js:9:67737 abort@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands_solution_wasm_bin.js:9:41049 _abort@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands_solution_wasm_bin.js:9:179948 wasm-stub@[wasm code] <?>.wasm-function[10471]@[wasm code] <?>.wasm-function[10466]@[wasm code] <?>.wasm-function[10461]@[wasm code] <?>.wasm-function[10458]@[wasm code] <?>.wasm-function[10474]@[wasm code] <?>.wasm-function[515]@[wasm code] <?>.wasm-function[502]@[wasm code] wasm-stub@[wasm code] [native code] SolutionWasm$send https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands.js:33:352 Q@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands.js:10:295 https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands.js:11:90 k@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands.js:22:322 promiseReactionJob@[native code] (evaluating 'new WebAssembly.RuntimeError(what)') (anonymous function) (hands_solution_wasm_bin.js:9:41099) promiseReactionJob

rahulscp commented 3 years ago

Not just in ios its not working in all mobiles i tested both android and Ios,

even the demo provided is also not working. is there any workaround ?

https://mediapipe.dev/demo/hands

Adolar13 commented 3 years ago

Same error in Hand demo not working on galaxy s7 and s10 in chrome browser. Firefox is working.

rahulscp commented 3 years ago

any update on this issue ?? still not working in all major mobiles

anderium commented 3 years ago

The Javascript API also does not work for Safari 14.0.2 (14610.3.7.1.10) on Mac OS X Mojave (10.14.6)

However, this might be unrelated as it only produces a single error in a different function than above. This is the error I'm getting:

The error in Safari on Mac OS X

It does work in Chrome on the same device.

jjeon5 commented 3 years ago

The Javascript API also does not work for Safari 14.0.2 (14610.3.7.1.10) on Mac OS X Mojave (10.14.6)

However, this might be unrelated as it only produces a single error in a different function than above. This is the error I'm getting:

The error in Safari on Mac OS X

It does work in Chrome on the same device.

I'm getting the same error on Safari.

wehriam commented 3 years ago

The texStorage2d method is not supported in Safari.

It appears the method is referenced in WASM builds such as @mediapipe/hands@0.1/hands_solution_wasm_bin.js.

There are a few references to the method:

@mcclanahoochie - Thank for you and your team's work on this project, and for responding to issues like this. Any updates on Mobile Safari support?

mhays-google commented 3 years ago

No update, but I will escalate this internally and have a better update soonish.

On Tue, Jan 19, 2021 at 8:22 AM John Wehr notifications@github.com wrote:

The texStorage2d method https://developer.mozilla.org/en-US/docs/Web/API/WebGL2RenderingContext/texStorage2D is not supported in Safari.

It appears the method is referenced in WASM builds such as @mediapipe/hands@0.1/hands_solution_wasm_bin.js https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands_solution_wasm_bin.js .

There are a few references to the method:

@mcclanahoochie https://github.com/mcclanahoochie - Thank for you and your team's work on this project, and for responding to issues like this https://github.com/google/mediapipe/issues/1227#issuecomment-717492010. Any updates on Mobile Safari support?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-762954648, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFK73JWLQXUB26I6AODS2WWVNANCNFSM4VHJHWYQ .

stevestavropoulos commented 3 years ago

Doesn't work on Chrome/Linux too. Firefox is the only working browser right now.

mhays-google commented 3 years ago

I'm updating the documentation -- right now this is developed for Chrome Mac/Windows (since that's what I can test on). I'm not deliberately excluding anything else, and I'll look into supporting it.

In the near future, I'm working on a lab to run this across a lot more devices, but I don't have the bandwidth right now.

On Thu, Feb 4, 2021 at 10:32 AM Steve Stavropoulos notifications@github.com wrote:

Doesn't work on Chrome/Linux too. Firefox is the only working browser right now.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-773517256, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFJF5YL4YYHQI242PGDS5LR2ZANCNFSM4VHJHWYQ .

stevestavropoulos commented 3 years ago

Also doesn't work on Chrome/Win10 and Edge/Win10. Firefox is working on both Windows and Linux.

wumpa-wumpa commented 3 years ago

For facemesh, on Android I get triple the framerate with mediapipe over tensorflow. Hopefully mediapipe on iOS would show similar performance gains and make mediapipe a robust solution for mobile webAR experiences. Any workarounds or idea of a timeline for iOS Safari?

tyrmullen commented 3 years ago

I think there are at least two separate issues here: (1) A lot of the incompatibility issues with Safari are related to WebGL2 support. Safari traditionally has only supported WebGL1, and their WebGL2 support is hidden behind a developer toggle. Furthermore, their WebGL2 support is incomplete, failing a large percentage of the official "conformance test" suite (https://www.khronos.org/registry/webgl/sdk/tests/webgl-conformance-tests.html).

As a workaround, many of our older web demos contain patches which will let them run on Safari when the experimental WebGL2 toggle is enabled, but some of the newer CodePens do not have similar patches yet. We were hoping to instead patch them so they run with WebGL1 alone and don't even need an experimental toggle. But for future versions of Safari, all of this may be unnecessary, as the latest Safari Technology Preview (https://developer.apple.com/safari/technology-preview/) appears to be aiming for full WebGL2 support built-in.

In any case, this is an area of active investigation, and Safari support should definitely be on the near horizon.

(2) Our initial ML inference was built for Desktop Chrome; on mobile devices (both iOS and Android), some of the stricter device limits cannot handle the original code paths and limits chosen. I see no reason why they shouldn't run on Linux, OSX and Windows Chrome, so if they're failing to run there, that seems like something to look into, and worth filing a separate issue for tracking.

For mobile, there are several possibilities we're looking into, so please bear with us on that front. I believe this is a similar issue to both https://github.com/google/mediapipe/issues/1430 and https://github.com/google/mediapipe/issues/1436.

narolski commented 3 years ago

@mhays-google Is there any update regarding this issue? Your demos are quite amazing, but the lack of the iOS/Safari support limits potential applications significantly 😞

akshatbjain commented 3 years ago

Hey guys! Any update on this issue? Really great work on building mediapipe. Unfortunately, it not working on iOS is a bit of a bummer. Would really appreciate it if you guys could take a look at solving this issue.

vKrypto commented 3 years ago

Mediapipe JS solution is being updated day by day, they are working on it..

till then you can use this code (3 march version )

https://www.ornaz.com/realtime/v0_3_march/

or find this code >>

var ASM_CONSTS = {393919: function() { in file hand_solution_wasm_bin.js and replace it with var ASM_CONSTS = {401815: function() {

or

puny-d commented 3 years ago

Hi, any update on this issue ?

nejurgis commented 3 years ago

Mediapipe JS solution is being updated day by day, they are working on it..

till then you can use this code (3 march version )

https://www.ornaz.com/realtime/v0_3_march/

or find this code >>

var ASM_CONSTS = {393919: function() { in file hand_solution_wasm_bin.js and replace it with var ASM_CONSTS = {401815: function() {

or

Unfortunately neither of the options you proposed works

mhays-google commented 3 years ago

We have an upcoming goal to grow our browser support, but it will definitely be later this year. As far as I know, however, Safari and Android Web are on top of that list.

On Wed, Apr 7, 2021 at 8:44 AM nejurgis @.***> wrote:

Mediapipe JS solution is being updated day by day, they are working on it..

till then you can use this code (3 march version )

https://www.ornaz.com/realtime/v0_3_march/

or find this code >>

var ASM_CONSTS = {393919: function() { in file hand_solution_wasm_bin.js and replace it with var ASM_CONSTS = {401815: function() {

or

Unfortunately neither of the options you proposed works

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-815019378, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFMHMWE2CFFYWA6AX2LTHR4VNANCNFSM4VHJHWYQ .

tyrmullen commented 3 years ago

Most of the groundwork has now been finished to allow JS Solutions to run on Desktop Safari-- you can preview these efforts in the "Holistic" demo, which I believe should now be runnable on Desktop Safari via WebGL 1. The other demos should work too, provided you're running on Solutions API version 0.3 or beyond (so in the "Hands" CodePen, replace any "hands@0.1" instance with "hands@0.3"; I believe there is one instance in the JS and several more in "Settings"-->"HTML"-->"Stuff for ").

Even for most of the failing CodePen demos, enabling experimental WebGL2 usually makes them work for me (running Safari 14.1 on an old MacBook Plus).

TL;DR: I believe Desktop Safari (and hence WebGL1 compatibility) is already (unofficially) supported. However, mobile support has not been looked into just yet, so there will likely still be some issues to work out in order to bring everything to iOS.

jadams777 commented 3 years ago

iOS Chrome and Safari support is critical for my MediaPipe Pose application. Would really appreciate your help adding support for these browsers. Thank you.

tyrmullen commented 3 years ago

One possible solution (at the potential cost of some performance) would be to run the ML CPU-side instead of GPU-side, since it's only GPU-side ML which encounters issues on iOS at the moment. We hope to have a nice option for this sort of toggling in a few weeks.

jadams777 commented 3 years ago

Hi, does this release include support for iOS? I don't see a changelog. Thank you. https://www.npmjs.com/package/@mediapipe/pose/v/0.4.1624666670

tyrmullen commented 3 years ago

No-- Desktop Safari is supported, but the toggling API I mentioned has not yet been finished, so iOS will still have issues. I will try to get a workaround patch into the next release. Will ping threads if/when I submit one.

peter-wildman commented 3 years ago

Is the iOS camera utils issue of not loading the camera a part of the known issues being looked at?

mhays-google commented 3 years ago

Yes it is. I've verified that I'm able to load the camera into iPad. Getting an iPhone to test.

On Sun, Jul 18, 2021 at 3:19 AM Peter Wildman @.***> wrote:

Is the iOS camera utils issue of not loading the camera a part of the known issues being looked at?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-882033768, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFMW7OXMYQRRARJDAO3TYKTEDANCNFSM4VHJHWYQ .

not-william commented 3 years ago

One possible solution (at the potential cost of some performance) would be to run the ML CPU-side instead of GPU-side, since it's only GPU-side ML which encounters issues on iOS at the moment. We hope to have a nice option for this sort of toggling in a few weeks.

Thanks for this suggestion. Would there be any way to do this currrently?

mhays-google commented 3 years ago

This is being rolled out now. It automatically detects iOS and it falls back onto CPU inference while we continue to troubleshoot openGL inference. This means things will run a bit slower right now, but the good news is that it should run. We are testing a few iPads and iPhones. The importance of iOS has been communicated.

On Tue, Jul 27, 2021 at 2:08 AM not-william @.***> wrote:

One possible solution (at the potential cost of some performance) would be to run the ML CPU-side instead of GPU-side, since it's only GPU-side ML which encounters issues on iOS at the moment. We hope to have a nice option for this sort of toggling in a few weeks.

Thanks for this suggestion. Would there be any way to do this currrently?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-887344426, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFMSUO5SAV7FE36ROSLTZZZPXANCNFSM4VHJHWYQ .

AmitMY commented 3 years ago

In the case it does work with WebGL on safari 15 (I did not check if it does), could you please only make the switch depending on the iOS safari version?

Source: https://developer.apple.com/documentation/safari-release-notes/safari-15-beta-release-notes image

mhays-google commented 3 years ago

The support for iOS will get better -- there is currently no exception made for Safari, whether on iOS or not. That is to say, Safari on Mac uses WebGL. We're discussing making this configurable. For now, the squeaky wheel is the iOS community at large, but I understand what you'd like and I'll address it soon.

On Thu, Jul 29, 2021 at 11:39 PM Amit Moryossef @.***> wrote:

In the case it does work with WebGL on safari 15 (I did not check if it does), could you please only make the switch depending on the iOS safari version?

Source:

https://developer.apple.com/documentation/safari-release-notes/safari-15-beta-release-notes [image: image] https://user-images.githubusercontent.com/5757359/127611415-d3e44e38-2d52-4bae-823c-5c5b7be13aa8.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-889666623, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFPDSOTU6LSG3TAP3E3T2JCIVANCNFSM4VHJHWYQ .

shubhamdawande commented 3 years ago

This is being rolled out now. It automatically detects iOS and it falls back onto CPU inference while we continue to troubleshoot openGL inference. This means things will run a bit slower right now, but the good news is that it should run. We are testing a few iPads and iPhones. The importance of iOS has been communicated.

@mhays-google Can we get an approximate release date of this workaround for facemesh? Current mediapipe provided solution is way more accurate than tfjs. Waiting on this release so that we can switch to mediapipe api.

mhays-google commented 3 years ago

It may actually already be up, but regardless, I'm pushing a new version before EOD today (Aug 2, PST).

On Mon, Aug 2, 2021 at 3:37 AM Shubham Dawande @.***> wrote:

This is being rolled out now. It automatically detects iOS and it falls back onto CPU inference while we continue to troubleshoot openGL inference. This means things will run a bit slower right now, but the good news is that it should run. We are testing a few iPads and iPhones. The importance of iOS has been communicated.

@mhays-google https://github.com/mhays-google Can we get an approximate release date of this workaround for facemesh? Current mediapipe provided solution is way more accurate than tfjs. Waiting on this release so that we can switch to mediapipe api.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-890920170, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFN5N3EKEIDH26YUKMTT2ZYPZANCNFSM4VHJHWYQ .

sanamumtaz commented 3 years ago

Kindly do share updates and link to demos that are working on iOS. (Does face_mesh demo work? Last time I checked on iOS, it didn't work.)

mhays-google commented 3 years ago

I just tested face_mesh on iOS running on Safari preview on my MacBook pro. It looks like it is working to me. I also tested face_mesh on iOS running on Safari on a modern iPad. It is working for me as well. Same with an iPad pro.

How did it not work? What version of iOS and OS are you using? What were you doing when it didn't work?

On Mon, Aug 16, 2021 at 5:30 AM Sana Mumtaz @.***> wrote:

Kindly do share updates and link to demos that are working on iOS. (Does face_mesh demo work? Last time I checked on iOS, it didn't work.)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-899471743, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFKPYN2NUUXXBB73MTLT5EAG5ANCNFSM4VHJHWYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

sanamumtaz commented 3 years ago

I tested on iPhone Safari. Latest updated version.

The demo stays in loading state. IMG-20210816-WA0014

ionif commented 3 years ago

Same for me on a iPhone 11 with iOS 14.6 (stays loading and never works). This is true for facemesh, and hands for me

On Mon, Aug 16, 2021 at 4:00 PM Sana Mumtaz @.***> wrote:

I tested on iPhone Safari. Latest updated version.

The demo stays in loading state. [image: IMG-20210816-WA0014] https://user-images.githubusercontent.com/35191030/129622462-9c05e06b-aaaf-4429-93fc-c94893a203e7.jpg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-899780539, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEXWUULKXFVD47GPIKRYDITT5FU6DANCNFSM4VHJHWYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

mhays-google commented 3 years ago

Hm. Thanks for this. You are right -- The demos in question were using an older version of controls. The new version specifically addresses an issue with ios phones. I've updated the versions to the latest and I've verified that all of the demos work on an iPhone 11. I think that the speed leaves something to be desired, but we are working on improving this. Also, I need to do something a bit different for selfie segmentation, so at least for now it's not really working.

Please try again and tell me how it goes.

On Mon, Aug 16, 2021 at 1:03 PM Alex Ionkov @.***> wrote:

Same for me on a iPhone 11 with iOS 14.6 (stays loading and never works). This is true for facemesh, and hands for me

On Mon, Aug 16, 2021 at 4:00 PM Sana Mumtaz @.***> wrote:

I tested on iPhone Safari. Latest updated version.

The demo stays in loading state. [image: IMG-20210816-WA0014] < https://user-images.githubusercontent.com/35191030/129622462-9c05e06b-aaaf-4429-93fc-c94893a203e7.jpg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/google/mediapipe/issues/1427#issuecomment-899780539 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AEXWUULKXFVD47GPIKRYDITT5FU6DANCNFSM4VHJHWYQ

. Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675

or Android < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-899782337, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFJCDFV5QQ42GZY4QYTT5FVHZANCNFSM4VHJHWYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

AmitMY commented 3 years ago

iPhone 11 Pro here. The codepen demo captures a single shot from my camera, and repeatedly runs pose estimation on it (not moving). This is consistent with both the front and back cameras. Performance - when I am in frame, 4-5 fps. out of frame, 9 fps. (tested in Safari and Chrome, iOS 15)

mhays-google commented 3 years ago

That's no good. I'm not getting the same issue. I'm bringing this up to the team to see if we can prioritize more iOS support.

On Mon, Aug 16, 2021 at 10:54 PM Amit Moryossef @.***> wrote:

iPhone 11 Pro here. The codepen demo captures a single shot from my camera, and repeatedly runs pose estimation on it (not moving). This is consistent with both the front and back cameras. Performance - when I am in frame, 4-5 fps. out of frame, 9 fps.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-900013297, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFNH3GR7OTMP4C7JZVLT5H2QNANCNFSM4VHJHWYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

jadams777 commented 3 years ago

Hi Michael, are you able to get the pose tracker running on iOS? I visited this codepen on my iPhone 12, but it didn't track my body. The controls cover the screen, so it's hard to see what's happening. Thanks.

https://codepen.io/mediapipe/pen/jOMbvxw

tyrmullen commented 3 years ago

At present, we're prioritizing functionality of the base API over performance and demo fixes, so no ETA for any of the other issues, but we'll be looking into fixing the selfie-segmentation API shortly.

Just to summarize, on most iPhones the current state of the demos should be the following:

** The "stays loading and never works" issue is usually this-- you can tell if this is the problem you're running into because the console log will give you a "Permissions denied" error, even though you accepted the camera permissions popup.

*** One workaround I applied lets them sometimes use 16-bit floating point textures rather than 32-bit ones, as iOS devices often don't support the WEBGL_color_buffer_float extension.

@jadams777 Pose seems to be working for me (at a few fps) on an iPhone 6S with Safari. I use the "full" demo so I can see more of the display: https://codepen.io/mediapipe/full/jOMbvxw, but I agree, it would be nice for iOS if we could scale back the controls (or maybe just add a toggle to show/hide those potentially?).

@AmitMY I have not seen this issue myself, but I've run into issues with similar symptoms in the past with iOS-Safari, where the pipeline is running, but the current video frame texture being passed back from the webcam stream isn't being updated by the browser. IIRC, this could happen when the browser thinks the video isn't being used, and so tries to throttle the rendering for it. As a test, perhaps try see if displaying the video directly on the webpage fixes the playback issue for you (although of course that will add an unnecessary extra rendering cost). And in any case, let's file this as a separate bug if it persists for you, since I believe it's unrelated to the other issues here.

jadams777 commented 3 years ago

Great idea RE adding a toggle to hide the controls :-)

On Wed, Aug 18, 2021 at 3:04 PM tyrmullen @.***> wrote:

At present, we're prioritizing functionality of the base API over performance and demo fixes, so no ETA for any of the other issues, but we'll be looking into fixing the selfie-segmentation API shortly.

Just to summarize, on most iPhones the current state of the demos should be the following:

  • Demos except selfie-segmentation should work*

  • Webcam should work with latest control_utils update**

  • Pre-canned videos (like for Objectron) should work

  • Uploading a video file might still fail for some devices

  • Technically there may be a slight loss in precision for mediapipe::Tensors when used on iOS, but in practice I've seen no negative impact as of yet***

  • We now automatically switch over to using CPU inference on iOS devices

  • The rendering is still being performed with WebGL1, as Safari still does not yet support WebGL2 (hopefully soon this may change?)

  • Performance is much slower, at least in part as a result of using CPU inference, but this has not been extensively investigated/profiled yet

  • There are quite a few calculators unique to selfie-segmentation which we need to ensure iOS-compatibility for; also it's the only API which outputs two textures.

** The "stays loading and never works" issue is usually this-- you can tell if this is the problem you're running into because the console log will give you a "Permissions denied" error, even though you accepted the camera permissions popup.

*** One workaround I applied lets them sometimes use 16-bit floating point textures rather than 32-bit ones, as iOS devices often don't support the WEBGL_color_buffer_float extension https://developer.mozilla.org/en-US/docs/Web/API/WEBGL_color_buffer_float .

@jadams777 https://github.com/jadams777 Pose seems to be working for me (at a few fps) on an iPhone 6S with Safari. I use the "full" demo so I can see more of the display: https://codepen.io/mediapipe/full/jOMbvxw, but I agree, it would be nice for iOS if we could scale back the controls (or maybe just add a toggle to show/hide those potentially?).

@AmitMY https://github.com/AmitMY I have not seen this issue myself, but I've run into issues with similar symptoms in the past with iOS-Safari, where the pipeline is running, but the current video frame texture being passed back from the webcam stream isn't being updated by the browser. IIRC, this could happen when the browser thinks the video isn't being used, and so tries to throttle the rendering for it. As a test, perhaps try see if displaying the video directly on the webpage fixes the playback issue for you (although of course that will add an unnecessary extra rendering cost). And in any case, let's file this as a separate bug if it persists for you, since I believe it's unrelated to the other issues here.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-901459813, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARDSKD4PBNRLTQCAGOM45Q3T5QVARANCNFSM4VHJHWYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

jadams777 commented 3 years ago

I tried the Pose codepen on my iPad Pro 2021, and there is a weird flashing effect. Check it out: https://photos.app.goo.gl/hAx5wRMavc2gTa4H8

AmitMY commented 3 years ago

The rendering is still being performed with WebGL1, as Safari still does not yet support WebGL2 (hopefully soon this may change?)

WebGL2 is available on iOS 15. Betas are currently available, public release in 1 month. https://caniuse.com/webgl2

peter-wildman commented 3 years ago

I've recently had this error emerge in Safari on Desktop. Is there a known solution:

Screen Shot 2021-08-20 at 1 51 23 pm
jadams777 commented 3 years ago

The latest MediaPipe update appears to have broken iOS support. The Pose code pen no longer works on my iPhone 12. https://codepen.io/mediapipe/full/jOMbvxw

On Thu, Aug 19, 2021 at 8:53 PM Peter Wildman @.***> wrote:

I've recently had this error emerge in Safari on Desktop. Is there a known solution: [image: Screen Shot 2021-08-20 at 1 51 23 pm] https://user-images.githubusercontent.com/6938599/130176567-4f7b81fb-160b-4aec-96b6-2a256ed27f9f.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-902410454, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARDSKD7NTSETLZQPMDNUB5DT5XGUXANCNFSM4VHJHWYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

mhays-google commented 3 years ago

Can you refresh your cache and try again? Console output would be useful. The update last night only exposed a new "VERSION" variable, but the demo is locked to @.***, so any time I push a new build version, there's still a small window of time where you are getting the old library, and in this case, there will be no VERSION symbol found.

On Fri, Aug 20, 2021 at 12:30 AM jadams777 @.***> wrote:

The latest MediaPipe update appears to have broken iOS support. The Pose code pen no longer works on my iPhone 12. https://codepen.io/mediapipe/full/jOMbvxw

On Thu, Aug 19, 2021 at 8:53 PM Peter Wildman @.***> wrote:

I've recently had this error emerge in Safari on Desktop. Is there a known solution: [image: Screen Shot 2021-08-20 at 1 51 23 pm] < https://user-images.githubusercontent.com/6938599/130176567-4f7b81fb-160b-4aec-96b6-2a256ed27f9f.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/google/mediapipe/issues/1427#issuecomment-902410454 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ARDSKD7NTSETLZQPMDNUB5DT5XGUXANCNFSM4VHJHWYQ

. Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675

or Android < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-902494501, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFKFMJLV2QSBLIIMOBLT5X77RANCNFSM4VHJHWYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

jadams777 commented 3 years ago

The latest Pose codepen is not working on any of my devices, actually. Here is a video using Chrome 92.0.4515.159 (64-bit): http://somup.com/crjYDLrcdR https://mailtrack.io/trace/link/065cf949bd365be3d57f53331722e094849b078a?url=http%3A%2F%2Fsomup.com%2FcrjYDLrcdR&userId=5826767&signature=cfa909eec5e10f47

Console output: jOMbvxw:1 Refused to apply style from ' https://cdpn.io/mediapipe/fullpage/demo.css https://mailtrack.io/trace/link/1b97c4a5dac84f21d468464d32454db939c5cb05?url=https%3A%2F%2Fcdpn.io%2Fmediapipe%2Ffullpage%2Fdemo.css&userId=5826767&signature=30e44dc2e657d0b5' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled. landmark_grid.ts:631 Uncaught ReferenceError: javascript is not defined at landmark_grid.ts:631 (anonymous) @ landmark_grid.ts:631 jOMbvxw:216 GET https://cdpn.io/mediapipe/fullpage/logo_white.png https://mailtrack.io/trace/link/433bd042f7165fa9b5448956eba5c12ac7719208?url=https%3A%2F%2Fcdpn.io%2Fmediapipe%2Ffullpage%2Flogo_white.png&userId=5826767&signature=e7733b58a4852d5c 404 landmark_grid.ts:631 GET https://cdpn.io/mediapipe/fullpage/deps.js https://mailtrack.io/trace/link/19cd544c2bb7ce740a43584a861eb3ec1097f099?url=https%3A%2F%2Fcdpn.io%2Fmediapipe%2Ffullpage%2Fdeps.js&userId=5826767&signature=36ea7c96b07ad3d7 net::ERR_ABORTED 404 goog.Dependency.load @ landmark_grid.ts:631 (anonymous) @ landmarkgrid.ts:631 goog.DebugLoader.loadDeps_ @ landmarkgrid.ts:631 goog.DebugLoader.loadClosureDeps @ landmark_grid.ts:631 (anonymous) @ landmark_grid.ts:631 pose_solution_simd_wasm_bin.js:9 Uncaught (in promise) TypeError: Cannot read property 'buffer' of undefined at Object.write (VM21 pose_solution_simd_wasm_bin.js:9) at Object.write (VM21 pose_solution_simd_wasm_bin.js:9) at Object.createDataFile (VM21 pose_solution_simd_wasm_bin.js:9) at finish (VM21 pose_solution_simd_wasm_bin.js:9) at processData (VM21 pose_solution_simd_wasm_bin.js:9) at Object.createPreloadedFile [as FS_createPreloadedFile] (VM21 pose_solution_simd_wasm_bin.js:9) at DataRequest.finish (VM20 pose_solution_packed_assets_loader.js:136) at DataRequest.onload (VM20 pose_solution_packed_assets_loader.js:131) at processPackageData (VM20 pose_solution_packed_assets_loader.js:167) at runWithFS (VM20 pose_solution_packed_assets_loader.js:178) write @ pose_solution_simd_wasm_bin.js:9 write @ pose_solution_simd_wasm_bin.js:9 createDataFile @ pose_solution_simd_wasm_bin.js:9 finish @ pose_solution_simd_wasm_bin.js:9 processData @ pose_solution_simd_wasm_bin.js:9 createPreloadedFile @ pose_solution_simd_wasm_bin.js:9 finish @ VM20 pose_solution_packed_assets_loader.js:136 onload @ VM20 pose_solution_packed_assets_loader.js:131 processPackageData @ VM20 pose_solution_packed_assets_loader.js:167 runWithFS @ VM20 pose_solution_packed_assets_loader.js:178 callRuntimeCallbacks @ VM21 pose_solution_simd_wasm_bin.js:9 preRun @ VM21 pose_solution_simd_wasm_bin.js:9 run @ VM21 pose_solution_simd_wasm_bin.js:9 runCaller @ VM21 pose_solution_simd_wasm_bin.js:9 removeRunDependency @ VM21 pose_solution_simd_wasm_bin.js:9 receiveInstance @ VM21 pose_solution_simd_wasm_bin.js:9 receiveInstantiationResult @ VM21 pose_solution_simd_wasm_bin.js:9 Promise.then (async) Y.tick @ control_utils.js:37 (anonymous) @ control_utils.js:36 requestAnimationFrame (async) Z @ control_utils.js:36 a.video.onloadedmetadata @ control_utils.js:37 pose_solution_simd_wasm_bin.js:9 still waiting on run dependencies: (anonymous) @ pose_solution_simd_wasm_bin.js:9 setInterval (async) addRunDependency @ pose_solution_simd_wasm_bin.js:9 createWasm @ pose_solution_simd_wasm_bin.js:9 (anonymous) @ pose_solution_simd_wasm_bin.js:9 (anonymous) @ pose.js:50 ta @ pose.js:14 next @ pose.js:15 (anonymous) @ pose.js:41 Y @ pose.js:41 (anonymous) @ pose.js:50 Promise.then (async) (anonymous) @ pose.js:50 ta @ pose.js:14 next @ pose.js:15 g @ pose.js:41 Promise.then (async) h @ pose.js:41 (anonymous) @ pose.js:41 Y @ pose.js:41 Lb @ pose.js:49 (anonymous) @ pose.js:58 ta @ pose.js:14 next @ pose.js:15 (anonymous) @ pose.js:41 Y @ pose.js:41 A.initialize @ pose.js:58 (anonymous) @ pose.js:59 ta @ pose.js:14 next @ pose.js:15 g @ pose.js:41 Promise.then (async) h @ pose.js:41 (anonymous) @ pose.js:41 Y @ pose.js:41 A.send @ pose.js:59 (anonymous) @ pose.js:77

tomhsiao1260 commented 3 years ago

Hi, everyone.

The current CodePen pose demo works well on my iPhone12 device. https://codepen.io/mediapipe/pen/jOMbvxw

However, I found that this demo use controls.SourcePicker in control_utils npm package to open the video. In some use cases, I think it's more convenient to use camera_utils instead to open the video like the following document does. https://google.github.io/mediapipe/solutions/pose#javascript-solution-api

However, after testing, it seems that camera_utils still cannot support IOS device (the latest npm update was 4 months ago). Just want to ask, are there any plans about the camera_utils IOS support for developers who don't need a control panel in their application?

mhays-google commented 3 years ago

camera_utils and control_utils were little helpers I wrote for the demos and are not in any way recommendations for the community. I don't know if we really even document them at mediapipe.dev.

However... Having said that... it's nice to have working things to help build stuff fast. :-)

To get camera utils working on iOS, you will need to add two attributes to the video element you are passing to the constructor.:

    crossorigin="anonymous"
    playsinline="true"

On Sun, Aug 22, 2021 at 9:19 PM Yao Hsiao @.***> wrote:

Hi, everyone.

The current CodePen pose demo works well on my iPhone12 device. https://codepen.io/mediapipe/pen/jOMbvxw

I found that this demo use controls.SourcePicker in control_utils npm package to open the video. However, in some use cases, it's more convenient to use camera_utils instead to open the video like the following document do. https://google.github.io/mediapipe/solutions/pose#javascript-solution-api

After testing, it seems that camera_utils still cannot support IOS device (the latest npm update was 4 months ago). Just want to ask, are there any plans about the camera_utils IOS support for developers who don't need a control panel in their application?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/mediapipe/issues/1427#issuecomment-903430536, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKQBFIPTFEQPCJADKGZYT3T6HD43ANCNFSM4VHJHWYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

tomhsiao1260 commented 3 years ago

Thanks for clarifying. Also, the solution works. Really appreciate!