search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.72k stars 5.18k forks source link

WebGL in webworker using Googles facemesh/mediapipe with WebGL backend and Offscreen canvas is broken in Chrome 93 #2528

Closed caspar-goeke closed 3 years ago

caspar-goeke commented 3 years ago

System information (Please provide as much relevant information as possible)

Describe the current behavior: After about running media-pipe/facemesh for about 2 minutes it crashes when using Offscreen Canvas. It's not a memory leak issue on our side (no tensors are leaking):

Describe the expected behavior: I would expect that it is possible to use a Offscreen canvas in a webworker that is running mediapipe/facemesh

Standalone code to reproduce the issue: We currently disabled the Offscreen canvas but otherwise the behavior can be reproduced here: https://www.labvanced.com/player.html?id=26130

Other info / Complete Logs :

sgowroji commented 3 years ago

Hi @caspar-goeke, Could you please verify issue is possible duplicate https://github.com/google/mediapipe/issues/2506

kenrussell commented 3 years ago

Could you please attach the contents of Chrome's about:gpu page ("Copy report to clipboard" / paste) as a text file here?

caspar-goeke commented 3 years ago

Graphics Feature Status

  • Canvas: Hardware accelerated
  • Compositing: Hardware accelerated
  • Multiple Raster Threads: Enabled
  • Out-of-process Rasterization: Hardware accelerated
  • OpenGL: Enabled
  • Rasterization: Hardware accelerated
  • Skia Renderer: Enabled
  • Video Decode: Hardware accelerated
  • Vulkan: Disabled
  • WebGL: Hardware accelerated
  • WebGL2: Hardware accelerated

Driver Bug Workarounds

  • clear_uniforms_before_first_program_use
  • decode_encode_srgb_for_generatemipmap
  • disable_delayed_copy_nv12
  • enable_webgl_timer_query_extensions
  • exit_on_context_lost
  • disabled_extension_GL_KHR_blend_equation_advanced
  • disabled_extension_GL_KHR_blend_equation_advanced_coherent

Problems Detected

ANGLE Features

  • allow_compressed_formats (Frontend workarounds): Enabled: true
    Allow compressed formats
  • disable_anisotropic_filtering (Frontend workarounds): Disabled
    Disable support for anisotropic filtering
  • disable_program_binary (Frontend features) anglebug:5007: Disabled
    Disable support for GL_OES_get_program_binary
  • disable_program_caching_for_transform_feedback (Frontend workarounds): Disabled
    On some GPUs, program binaries don't contain transform feedback varyings
  • enableCompressingPipelineCacheInThreadPool (Frontend workarounds) anglebug:4722: Disabled: false
    Enable compressing pipeline cache in thread pool.
  • enable_capture_limits (Frontend features) anglebug:5750: Disabled
    Set the context limits like frame capturing was enabled
  • forceRobustResourceInit (Frontend workarounds) anglebug:6041: Disabled
    Force-enable robust resource init
  • lose_context_on_out_of_memory (Frontend workarounds): Enabled: true
    Some users rely on a lost context notification if a GL_OUT_OF_MEMORY error occurs
  • scalarize_vec_and_mat_constructor_args (Frontend workarounds) 1165751: Disabled: false
    Always rewrite vec/mat constructors to be consistent
  • sync_framebuffer_bindings_on_tex_image (Frontend workarounds): Disabled
    On some drivers TexImage sometimes seems to interact with the Framebuffer
  • add_mock_texture_no_render_target (D3D workarounds) anglebug:2152: Disabled: isIntel && capsVersion < IntelDriverVersion(4815)
    On some drivers when rendering with no render target, two bugs lead to incorrect behavior
  • allow_clear_for_robust_resource_init (D3D workarounds) 941620: Enabled: true
    Some drivers corrupt texture data when clearing for robust resource initialization.
  • allow_translate_uniform_block_to_structured_buffer (D3D workarounds) anglebug:3682: Enabled: IsWin10OrGreater()
    There is a slow fxc compile performance issue with dynamic uniform indexing if translating a uniform block with a large array member to cbuffer.
  • call_clear_twice (D3D workarounds) 655534: Disabled: isIntel && isSkylake && capsVersion < IntelDriverVersion(4771)
    Using clear() may not take effect
  • depth_stencil_blit_extra_copy (D3D workarounds) anglebug:1452: Disabled: (part1 <= 13u && part2 < 6881) && isNvidia && driverVersionValid
    Bug in some drivers triggers a TDR when using CopySubresourceRegion from a staging texture to a depth/stencil
  • disable_b5g6r5_support (D3D workarounds): Disabled: (isIntel && capsVersion < IntelDriverVersion(4539)) || isAMD
    Textures with the format DXGI_FORMAT_B5G6R5_UNORM have incorrect data
  • emulate_isnan_float (D3D workarounds) 650547: Disabled: isIntel && isSkylake && capsVersion < IntelDriverVersion(4542)
    Using isnan() on highp float will get wrong answer
  • emulate_tiny_stencil_textures (D3D workarounds): Disabled: isAMD && !(deviceCaps.featureLevel < D3D_FEATURE_LEVEL_10_1)
    1x1 and 2x2 mips of depth/stencil textures aren't sampled correctly
  • expand_integer_pow_expressions (D3D workarounds): Enabled: true
    The HLSL optimizer has a bug with optimizing 'pow' in certain integer-valued expressions
  • flush_after_ending_transform_feedback (D3D workarounds): Enabled: isNvidia
    Some drivers sometimes write out-of-order results to StreamOut buffers when transform feedback is used to repeatedly write to the same buffer positions
  • force_atomic_value_resolution (D3D workarounds) anglebug:3246: Enabled: isNvidia
    On some drivers the return value from RWByteAddressBuffer.InterlockedAdd does not resolve when used in the .yzw components of a RWByteAddressBuffer.Store operation
  • get_dimensions_ignores_base_level (D3D workarounds): Enabled: isNvidia
    Some drivers do not take into account the base level of the texture in the results of the HLSL GetDimensions builtin
  • mrt_perf_workaround (D3D workarounds): Enabled: true
    Some drivers have a bug where they ignore null render targets
  • pre_add_texel_fetch_offsets (D3D workarounds): Disabled: isIntel
    HLSL's function texture.Load returns 0 when the parameter Location is negative, even if the sum of Offset and Location is in range
  • rewrite_unary_minus_operator (D3D workarounds): Disabled: isIntel && (isBroadwell || isHaswell) && capsVersion < IntelDriverVersion(4624)
    Evaluating unary minus operator on integer may get wrong answer in vertex shaders
  • select_view_in_geometry_shader (D3D workarounds): Disabled: !deviceCaps.supportsVpRtIndexWriteFromVertexShader
    The viewport or render target slice will be selected in the geometry shader stage for the ANGLE_multiview extension
  • set_data_faster_than_image_upload (D3D workarounds): Enabled: !(isIvyBridge || isBroadwell || isHaswell)
    Set data faster than image upload
  • skip_vs_constant_register_zero (D3D workarounds): Enabled: isNvidia
    In specific cases the driver doesn't handle constant register zero correctly
  • use_instanced_point_sprite_emulation (D3D workarounds): Disabled: isFeatureLevel9_3
    Some D3D11 renderers do not support geometry shaders for pointsprite emulation
  • use_system_memory_for_constant_buffers (D3D workarounds) 593024: Disabled: isIntel
    Copying from staging storage to constant buffer storage does not work
  • zero_max_lod (D3D workarounds): Disabled: isFeatureLevel9_3
    Missing an option to disable mipmaps on a mipmapped texture

Version Information

Data exported | 2021-09-15T03:45:05.541Z -- | -- Chrome version | Chrome/93.0.4577.63 Operating system | Windows NT 10.0.22454 Software rendering list URL | https://chromium.googlesource.com/chromium/src/+/ff5c0da2ec0adeaed5550e6c7e98417dac77d98a/gpu/config/software_rendering_list.json Driver bug list URL | https://chromium.googlesource.com/chromium/src/+/ff5c0da2ec0adeaed5550e6c7e98417dac77d98a/gpu/config/gpu_driver_bug_list.json ANGLE commit id | f42bd00efd49 2D graphics backend | Skia/93 42c2cee5ce92a792e79ec22a5a20f8c57e32892b Command Line | "C:\Program Files\Google\Chrome\Application\chrome.exe" --flag-switches-begin --flag-switches-end

Log Messages

  • GpuProcessHost: The info collection GPU process exited normally. Everything is okay.
  • GpuProcessHost: The info collection GPU process exited normally. Everything is okay.
caspar-goeke commented 3 years ago

@sgowroji the issue seems to be related, but it's hard to say if its a douplicate. We do not run the holistic part but just the facemesh, and the actual error message seems different, so my guess is it's not idential. However, what seems to be similar is that in Chrome 93 we cannot use an offscreen canvas on the webworker and then feed an image based on that into mediapipe to get a tensor which then does the face detection, while this was a workable solution in Chrome 92. The speed reduction not using an offscreen canvas but image data directly is about 20-40% which is really a lot for our application. We are happy for any suggestions (obviously we are also trying oursevels to find any workaround).

caspar-goeke commented 3 years ago

@kenrussell please find the output of my graphics card detials above, but please also note that this happens on each and every system / graphics card we tested so far (about 10 different ones), and both on Linux and Windows

caspar-goeke commented 3 years ago

Even some more information if needed

Initialization time | 192 In-process GPU | false Passthrough Command Decoder | true Sandboxed | true GPU0 | VENDOR= 0x10de, DEVICE=0x2484, SUBSYS=0x87bd1043, REV=161, LUID={0,43502} *ACTIVE* GPU1 | VENDOR= 0x1414, DEVICE=0x008c, LUID={0,46645} Optimus | false AMD switchable | false Desktop compositing | Aero Glass Direct composition | true Supports overlays | true YUY2 overlay support | DIRECT NV12 overlay support | DIRECT BGRA8 overlay support | SOFTWARE RGB10A2 overlay support | SOFTWARE Driver D3D12 feature level | D3D 12.2 Driver Vulkan API version | Vulkan API 1.2.0 Driver vendor | NVIDIA Driver version | 30.0.14.7141 GPU CUDA compute capability major version | 0 Pixel shader version | 5.0 Vertex shader version | 5.0 Max. MSAA samples | 8 Machine model name |   Machine model version |   GL_VENDOR | Google Inc. (NVIDIA) GL_RENDERER | ANGLE (NVIDIA, NVIDIA GeForce RTX 3070 Direct3D11 vs_5_0 ps_5_0, D3D11-30.0.14.7141) GL_VERSION | OpenGL ES 2.0.0 (ANGLE 2.1.16203 git hash: f42bd00efd49)

tyrmullen commented 3 years ago

As mentioned on the Chromium bug, I think it's likely that this is not a duplicate issue, but also not a MediaPipe issue.

google-ml-butler[bot] commented 3 years ago

Are you satisfied with the resolution of your issue? Yes No