FaceMesh Javascript iOS Runtime Errors

[merrijo]

Hi all,

We're trying to use FaceMesh for a mobile webapp, and are struggling to get it working for iOS. Android works great for Chrome, haven't tested in Firefox yet. It looks like the same error stack from #1427, and looking at logs via a Mac PC we can see the following Warnings/Errors:

[Warning] I0527 11:52:48.321000       1 gl_context_webgl.cc:119] Couldn't create webGL 2 context. (face_mesh_solution_wasm_bin.js, line 9)
[Warning] W0527 11:52:48.322000       1 gl_context_webgl.cc:144] Creating a context with WebGL 2 failed: UNKNOWN: emscripten_webgl_create_context() returned error 0 (face_mesh_solution_wasm_bin.js, line 9)
[Warning] W0527 11:52:48.323000       1 gl_context_webgl.cc:145] Fall back on WebGL 1. (face_mesh_solution_wasm_bin.js, line 9)
[Warning] I0527 11:52:48.324000       1 gl_context_webgl.cc:150] Successfully created a WebGL context with major version 2 and handle 3 (face_mesh_solution_wasm_bin.js, line 9)
[Warning] I0527 11:52:48.326000       1 gl_context.cc:348] GL version: 2.0 (OpenGL ES 2.0 (WebGL 1.0)) (face_mesh_solution_wasm_bin.js, line 9)
[Warning] W0527 11:52:48.328000       1 gl_context.cc:807] OpenGL error checking is disabled (face_mesh_solution_wasm_bin.js, line 9)
[Warning] INFO: Replacing 164 node(s) with delegate (unknown) node, yielding 1 partitions. (face_mesh_solution_wasm_bin.js, line 9)
[Warning] INFO: Replacing 97 node(s) with delegate (unknown) node, yielding 1 partitions. (face_mesh_solution_wasm_bin.js, line 9)
[Warning] E0527 11:52:48.822000       1 calculator_graph.cc:772] INTERNAL: CalculatorGraph::Run() failed in Run:  (face_mesh_solution_wasm_bin.js, line 9)
[Warning] Calculator::Open() for node "facelandmarkfrontgpu_facelandmarkgpuinferencecalculatorfacelandmarkfrontgpufacelandmarkgpuInferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\xbd\x02\x08\r\x12\xb8\x02\x43\x61lculator::Open() for node \"facelandmarkfrontgpufacelandmarkgpuinferencecalculatorfacelandmarkfrontgpufacelandmarkgpu_InferenceCalculator\" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50'] (face_mesh_solution_wasm_bin.js, line 9)
[Warning] F0527 11:52:48.824000       1 solutions_wasm.embind.cc:841] Check failed: absl::OkStatus() == (graph_->WaitUntilIdle()) (OK vs. INTERNAL: CalculatorGraph::Run() failed in Run:  (face_mesh_solution_wasm_bin.js, line 9)
[Warning] Calculator::Open() for node "facelandmarkfrontgpu_facelandmarkgpuinferencecalculatorfacelandmarkfrontgpufacelandmarkgpuInferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\xbd\x02\x08\r\x12\xb8\x02\x43\x61lculator::Open() for node \"facelandmarkfrontgpufacelandmarkgpuinferencecalculatorfacelandmarkfrontgpufacelandmarkgpu_InferenceCalculator\" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50'])  (face_mesh_solution_wasm_bin.js, line 9)
[Warning] * Check failure stack trace: * (face_mesh_solution_wasm_bin.js, line 9)
[Warning] undefined (face_mesh_solution_wasm_bin.js, line 9)
[Error] Unhandled Promise Rejection: RuntimeError: abort(undefined) at jsStackTrace@https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh@0.3/face_mesh_solution_wasm_bin.js:9:64194
stackTrace@https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh@0.3/face_mesh_solution_wasm_bin.js:9:64449
abort@https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh@0.3/face_mesh_solution_wasm_bin.js:9:37802
_abort@https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh@0.3/face_mesh_solution_wasm_bin.js:9:176992
wasm-stub@[wasm code]
<?>.wasm-function[8770]@[wasm code]
<?>.wasm-function[8766]@[wasm code]
<?>.wasm-function[8762]@[wasm code]
<?>.wasm-function[8759]@[wasm code]
<?>.wasm-function[8773]@[wasm code]
<?>.wasm-function[256]@[wasm code]
<?>.wasm-function[194]@[wasm code]
wasm-stub@[wasm code]
[native code]
SolutionWasm$send
https://192.168.0.190:5001/bundle.js:45791:278
T@https://192.168.0.190:5001/bundle.js:45761:40
https://192.168.0.190:5001/bundle.js:45762:90
k@https://192.168.0.190:5001/bundle.js:45771:322
promiseReactionJob@[native code] (evaluating 'new WebAssembly.RuntimeError(what)')
    generatorResume (face_mesh_solution_wasm_bin.js:9:37852)
    rejected (bundle.js:130636)
    promiseReactionJob

We tried to enable WebGL 2.0 but it didn't seem to work. Is there a way to get this running off the CPU instead?

Edit: We quickly tried to enable WebGPU as well, but it looks like the iPhone we're testing with doesn't support FP16.

[sgowroji]

Duplicate https://github.com/google/mediapipe/issues/1427

[sgowroji]

Hi @joshmer, Have a look at this for your reference.

[merrijo]

@sgowroji Thanks for that.

However, as stated above WebGL 2.0 did not work in our case. According to the warning messages I posted, the iPhone failed to create a WebGL 2 context so it fell back to WebGL 1.0. We also tried WebGPU, but it also failed for a different reason -- FP16 not supported.

We tested on both iPhone 7 and iPhone X, and got the same results. I'm not sure if it matters, but we're not testing with the CodePen site as in #1427.

[tyrmullen]

iOS has never been supported for the JS Solution APIs (the same is true of Android Chrome for that matter, but fortuitously it often "just works"-- at least for 64-bit Android IIUC). However, iOS support is a feature we hope to add soon, and everything already runs on iOS-Safari except for the GPU-side ML. So one relatively easy workaround is to just use CPU-side ML (at the possible cost of some performance). In a few weeks, we hope to have the ability to toggle nicely between GPU-side and CPU-side ML, to at least allow for this workaround.

[merrijo]

Thanks for the update, that's unfortunate to hear.

[tyrmullen]

@joshmer Note that this issue has been closed as a duplicate of #1427, not because we consider it fixed yet. We do plan to support iOS in the future, but will use that issue for tracking.

Also, WebGPU would require a lot of separate backend infra code (all of the rendering/GPU code currently is written targeting WebGL), so WebGPU is not officially supported anywhere in our codebase at this time (and it is unlikely to be supported until Chrome has at least an Origin Trial for it).

[simlevesque]

iOS has never been supported for the JS Solution APIs (the same is true of Android Chrome for that matter, but fortuitously it often "just works"-- at least for 64-bit Android IIUC).

@tyrmullen Is it possible to add this to the documentation ?

[allanchua101]

iOS has never been supported for the JS Solution APIs (the same is true of Android Chrome for that matter, but fortuitously it often "just works"-- at least for 64-bit Android IIUC). However, iOS support is a feature we hope to add soon, and everything already runs on iOS-Safari except for the GPU-side ML. So one relatively easy workaround is to just use CPU-side ML (at the possible cost of some performance). In a few weeks, we hope to have the ability to toggle nicely between GPU-side and CPU-side ML, to at least allow for this workaround.

Do you have any guides on how to utilize the CPU-based* ML for JS-based solutions guys?