ARM Mali G51 GPU

[natarajakm]

Hi I am using custom arm based board with Mali G51 GPU. GPU spec available @ https://developer.arm.com/ip-products/graphics-and-multimedia/mali-gpus/mali-g51-gpu so we cross compiled desktop version of mediapipe to our board with openCV. the issue is, inference is failing. same code works on desktop with Nvidia Graphics card. Mali G51 GPU is support OpenGL ES 3.2. I tested handtracking CPU version sample it works fine. but GPU version doesn't. is there any other requirement this GPU is lacking ?

[sgowroji]

Hi @natarajakm, Could you please add the failing logs to understand the issue much better. Thanks!

[natarajakm]

hi @sgowroji, it is not crashing or failing in final output annotation are not drawing

after debug i noticed in tensors_to_detections_calculator.cc ... in this function

detectionscores[i] < options.min_score_thresh()) {

for every frame detection_scores is less than min_score_thresh so is continue from here. because of this in output plain video only appears not annotation marks. is it My GPU capable to run mediapipe ? apart from cross compile, not done any changes to original handtracking demo code

====> absl::Status TensorsToDetectionsCalculator::ConvertToDetections( const float detection_boxes, const float detection_scores, const int detection_classes, std::vector output_detections) { for (int i = 0; i < numboxes; ++i) { if (options_.has_min_score_thresh() && detectionscores[i] < options.min_score_thresh()) { continue; } const int box_offset = i numcoords; Detection detection = ConvertToDetection( detection_boxes[box_offset + 0], detection_boxes[box_offset + 1], detection_boxes[box_offset + 2], detection_boxes[box_offset + 3], detection_scores[i], detectionclasses[i], options.flip_vertically()); const auto& bbox = detection.location_data().relative_boundingbox(); if (bbox.width() < 0 || bbox.height() < 0) { // Decoded detection boxes could have negative values for width/height due // to model prediction. Filter out those boxes since some downstream // calculators may assume non-negative values. (b/171391719) continue; } // Add keypoints. if (options.num_keypoints() > 0) { auto location_data = detection.mutable_location_data(); for (int kp_id = 0; kpid < options.numkeypoints() * options.num_values_per_keypoint(); kpid += options.num_values_per_keypoint()) { auto keypoint = location_data->add_relative_keypoints(); const int keypoint_index = boxoffset + options.keypoint_coord_offset() + kp_id; keypoint->set_x(detection_boxes[keypoint_index + 0]); keypoint->sety(options.flip_vertically() ? 1.f - detection_boxes[keypoint_index + 1] : detection_boxes[keypoint_index + 1]); } } output_detections->emplace_back(detection); } return absl::OkStatus(); } =====>

[natarajakm]

this my GPUs OpenGl ES extentions list,

I think it supports float16 textures ... is this a problem

I20200101 00:29:52.946300 12987 gl_context.cc:253] GL_ARM_rgba8 I20200101 00:29:52.946323 12987 gl_context.cc:253] GL_ARM_mali_shader_binary I20200101 00:29:52.946341 12987 gl_context.cc:253] GL_OES_depth24 I20200101 00:29:52.946357 12987 gl_context.cc:253] GL_OES_depth_texture I20200101 00:29:52.946374 12987 gl_context.cc:253] GL_OES_depth_texture_cube_map I20200101 00:29:52.946393 12987 gl_context.cc:253] GL_OES_packed_depth_stencil I20200101 00:29:52.946410 12987 gl_context.cc:253] GL_OES_rgb8_rgba8 I20200101 00:29:52.946429 12987 gl_context.cc:253] GL_EXT_read_format_bgra I20200101 00:29:52.946446 12987 gl_context.cc:253] GL_OES_compressed_paletted_texture I20200101 00:29:52.946465 12987 gl_context.cc:253] GL_OES_compressed_ETC1_RGB8_texture I20200101 00:29:52.946483 12987 gl_context.cc:253] GL_OES_standard_derivatives I20200101 00:29:52.946499 12987 gl_context.cc:253] GL_OES_EGL_image I20200101 00:29:52.946516 12987 gl_context.cc:253] GL_OES_EGL_image_external I20200101 00:29:52.946535 12987 gl_context.cc:253] GL_OES_EGL_image_external_essl3 I20200101 00:29:52.946552 12987 gl_context.cc:253] GL_OES_EGL_sync I20200101 00:29:52.946568 12987 gl_context.cc:253] GL_OES_texture_npot I20200101 00:29:52.946585 12987 gl_context.cc:253] GL_OES_vertex_half_float I20200101 00:29:52.946601 12987 gl_context.cc:253] GL_OES_required_internalformat I20200101 00:29:52.946616 12987 gl_context.cc:253] GL_OES_vertex_array_object I20200101 00:29:52.946635 12987 gl_context.cc:253] GL_OES_mapbuffer I20200101 00:29:52.946652 12987 gl_context.cc:253] GL_EXT_texture_format_BGRA8888 I20200101 00:29:52.946668 12987 gl_context.cc:253] GL_EXT_texture_rg I20200101 00:29:52.946686 12987 gl_context.cc:253] GL_EXT_texture_type_2_10_10_10_REV I20200101 00:29:52.946704 12987 gl_context.cc:253] GL_OES_fbo_render_mipmap I20200101 00:29:52.946720 12987 gl_context.cc:253] GL_OES_element_index_uint I20200101 00:29:52.946738 12987 gl_context.cc:253] GL_EXT_shadow_samplers I20200101 00:29:52.946758 12987 gl_context.cc:253] GL_OES_texture_compression_astc I20200101 00:29:52.946774 12987 gl_context.cc:253] GL_KHR_texture_compression_astc_ldr I20200101 00:29:52.946791 12987 gl_context.cc:253] GL_KHR_texture_compression_astc_hdr I20200101 00:29:52.946807 12987 gl_context.cc:253] GL_KHR_texture_compression_astc_sliced_3d I20200101 00:29:52.946826 12987 gl_context.cc:253] GL_EXT_texture_compression_astc_decode_mode I20200101 00:29:52.946851 12987 gl_context.cc:253] GL_KHR_debug I20200101 00:29:52.946869 12987 gl_context.cc:253] GL_EXT_occlusion_query_boolean I20200101 00:29:52.946888 12987 gl_context.cc:253] GL_EXT_disjoint_timer_query I20200101 00:29:52.946907 12987 gl_context.cc:253] GL_EXT_blend_minmax I20200101 00:29:52.946924 12987 gl_context.cc:253] GL_EXT_discard_framebuffer I20200101 00:29:52.946943 12987 gl_context.cc:253] GL_OES_get_program_binary I20200101 00:29:52.946961 12987 gl_context.cc:253] GL_OES_texture_3D I20200101 00:29:52.946979 12987 gl_context.cc:253] GL_EXT_texture_storage I20200101 00:29:52.946997 12987 gl_context.cc:253] GL_EXT_multisampled_render_to_texture I20200101 00:29:52.947062 12987 gl_context.cc:253] GL_OES_surfaceless_context I20200101 00:29:52.947082 12987 gl_context.cc:253] GL_OES_texture_stencil8 I20200101 00:29:52.947101 12987 gl_context.cc:253] GL_EXT_shader_pixel_local_storage I20200101 00:29:52.947119 12987 gl_context.cc:253] GL_ARM_shader_framebuffer_fetch I20200101 00:29:52.947139 12987 gl_context.cc:253] GL_ARM_shader_framebuffer_fetch_depth_stencil I20200101 00:29:52.947156 12987 gl_context.cc:253] GL_ARM_mali_program_binary I20200101 00:29:52.947175 12987 gl_context.cc:253] GL_EXT_sRGB I20200101 00:29:52.947192 12987 gl_context.cc:253] GL_EXT_sRGB_write_control I20200101 00:29:52.947211 12987 gl_context.cc:253] GL_EXT_texture_sRGB_decode I20200101 00:29:52.947229 12987 gl_context.cc:253] GL_EXT_texture_sRGB_R8 I20200101 00:29:52.947247 12987 gl_context.cc:253] GL_EXT_texture_sRGB_RG8 I20200101 00:29:52.947265 12987 gl_context.cc:253] GL_KHR_blend_equation_advanced I20200101 00:29:52.947284 12987 gl_context.cc:253] GL_KHR_blend_equation_advanced_coherent I20200101 00:29:52.947302 12987 gl_context.cc:253] GL_OES_texture_storage_multisample_2d_array I20200101 00:29:52.947321 12987 gl_context.cc:253] GL_OES_shader_image_atomic I20200101 00:29:52.947340 12987 gl_context.cc:253] GL_EXT_robustness I20200101 00:29:52.947357 12987 gl_context.cc:253] GL_EXT_draw_buffers_indexed I20200101 00:29:52.947376 12987 gl_context.cc:253] GL_OES_draw_buffers_indexed I20200101 00:29:52.947393 12987 gl_context.cc:253] GL_EXT_texture_border_clamp I20200101 00:29:52.947412 12987 gl_context.cc:253] GL_OES_texture_border_clamp I20200101 00:29:52.947429 12987 gl_context.cc:253] GL_EXT_texture_cube_map_array I20200101 00:29:52.947448 12987 gl_context.cc:253] GL_OES_texture_cube_map_array I20200101 00:29:52.947466 12987 gl_context.cc:253] GL_OES_sample_variables I20200101 00:29:52.947484 12987 gl_context.cc:253] GL_OES_sample_shading I20200101 00:29:52.947502 12987 gl_context.cc:253] GL_OES_shader_multisample_interpolation I20200101 00:29:52.947521 12987 gl_context.cc:253] GL_EXT_shader_io_blocks I20200101 00:29:52.947540 12987 gl_context.cc:253] GL_OES_shader_io_blocks I20200101 00:29:52.947558 12987 gl_context.cc:253] GL_EXT_tessellation_shader I20200101 00:29:52.947576 12987 gl_context.cc:253] GL_OES_tessellation_shader I20200101 00:29:52.947594 12987 gl_context.cc:253] GL_EXT_primitive_bounding_box I20200101 00:29:52.947613 12987 gl_context.cc:253] GL_OES_primitive_bounding_box I20200101 00:29:52.947631 12987 gl_context.cc:253] GL_EXT_geometry_shader I20200101 00:29:52.947649 12987 gl_context.cc:253] GL_OES_geometry_shader I20200101 00:29:52.947669 12987 gl_context.cc:253] GL_ANDROID_extension_pack_es31a I20200101 00:29:52.947686 12987 gl_context.cc:253] GL_EXT_gpu_shader5 I20200101 00:29:52.947705 12987 gl_context.cc:253] GL_OES_gpu_shader5 I20200101 00:29:52.947723 12987 gl_context.cc:253] GL_EXT_texture_buffer I20200101 00:29:52.947741 12987 gl_context.cc:253] GL_OES_texture_buffer I20200101 00:29:52.947758 12987 gl_context.cc:253] GL_EXT_copy_image I20200101 00:29:52.947777 12987 gl_context.cc:253] GL_OES_copy_image I20200101 00:29:52.947808 12987 gl_context.cc:253] GL_EXT_shader_non_constant_global_initializers I20200101 00:29:52.947826 12987 gl_context.cc:253] GL_EXT_color_buffer_half_float I20200101 00:29:52.947845 12987 gl_context.cc:253] GL_EXT_color_buffer_float I20200101 00:29:52.947865 12987 gl_context.cc:253] GL_EXT_YUV_target I20200101 00:29:52.947882 12987 gl_context.cc:253] GL_OVR_multiview I20200101 00:29:52.947901 12987 gl_context.cc:253] GL_OVR_multiview2 I20200101 00:29:52.947918 12987 gl_context.cc:253] GL_OVR_multiview_multisampled_render_to_texture I20200101 00:29:52.947937 12987 gl_context.cc:253] GL_KHR_robustness I20200101 00:29:52.947957 12987 gl_context.cc:253] GL_KHR_robust_buffer_access_behavior I20200101 00:29:52.947975 12987 gl_context.cc:253] GL_EXT_draw_elements_base_vertex I20200101 00:29:52.947993 12987 gl_context.cc:253] GL_OES_draw_elements_base_vertex I20200101 00:29:52.948014 12987 gl_context.cc:253] GL_EXT_texture_filter_anisotropic

[natarajakm]

any update on this it will be grate. it will helps me to procced with this GPU or choose the other GPU option

[sgowroji]

Hi @natarajakm, Did you get chance to check the gpu support from here https://google.github.io/mediapipe/getting_started/gpu_support.html#gpu-support

[natarajakm]

Hi @sgowroji , yes i checked it it supports all OpenGL requirements of mediapipe. only difference is we have ARM Mali G51 GPU. OpenGL® ES 1.1, 2.0, 3.1, 3.2 Vulkan 1.0* OpenCL™ 1.1, 1.2, 2.0 Full Profile, RenderScript for more info: https://developer.arm.com/ip-products/graphics-and-multimedia/mali-gpus/mali-g51-gpu

it is embedded SoC have Cortex-A9 with Mali G51 GPU. we also have one more SoC have Coretex-A9 with PowerVR GPU. it works fine here.

After comparing both GPU's capabilities my observation is

1. Texture with FLOAT16 - in Mali GPU
2. Texture with FLOAT32 - in PowerVR GPU.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[natarajakm]

I would like to know google media supports ARM Mali G51 GPU or not. I provided GPU capability https://google.github.io/mediapipe/getting_started/gpu_support.html#gpu-support

[sgowroji]

Hi @natarajakm, We have not tried anything with ARM Mali G51 GPU yet. But there are some users who could able to execute it with similar GPU's https://github.com/google/mediapipe/issues/1218.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[sgowroji]

Hi @natarajakm, Please respond if you are still looking resolution for the above query. And did you get a chance to go through the above comment. Thanks

[natarajakm]

@sgowroji: yes still I am facing issue. What I understood from my R&D on the above issue. media pipe will not works on GPU s which don't have FLOAT32 support.

[mcclanahoochie]

any gpu that supports OpenGL ES 3.1+ should be able to run mediapipe, so it should work for this GPU. i'm not sure what is going on here, but i don't have that exact gpu to test.

based on https://github.com/google/mediapipe/issues/2332#issuecomment-908892531, what version of mediapipe are you running? have you tried the latest release at https://github.com/google/mediapipe/releases ?

the min_score_thresh you mention can be controlled here. it can be lowered, but you shouldn't have to do that. what range are the scores you are seeing?

Mali G51 gpu supports both fp16 and fp32 operations (benchmarks here), i don't think that is an issue.

the hand tracking demo runs two models: detection and landmarks. can you see if just the detection demo works? this can help narrow down where things go wrong.

also, do any other mediapipe GPU demos work? like selfie segmentation or object detection?

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No