juntao commented 1 year ago

Motivation

The mediapipe-rs project provides a Rust SDK to support mediapipe AI models. The SDK provides utility functions to pre-process application data (such as images, audio and video) into TFLite / PyTorch formats, and convert the inference results back into application data. In order to accomplish this, the mediapipe-rs project has made extensive use of the ffmpeg library. It compiles ffmpeg to Wasm and then builds it together with the application binary.

However, the issue with this approach is that those Wasm-compiled ffmpeg functions are slow. We believe a better approach is to create a ffmpeg plugin for WasmEdge, and allow Wasm applications to call native ffmpeg functions as host functions.

Expected outcome

The deliverables will be

A WasmEdge plugin for ffmpeg that is similar to the WasmEdge OpenCV-mini plugin. That is to re-export ffmpeg functions in C style as plugin functions as covered in the plugin developer guide.
A Rust SDK for Wasm programs to access ffmpeg functions in the plugin. Similar to the WasmEdge OpenCV-mini SDK
Refactor the mediapipe-rs project to use the ffmpeg plugin instead of compiling ffmpeg to Wasm.

We expect the deliverable will cover at least 80% of all ffmpeg functions, and have complete unit test and documentation coverage.

Recommended Skills: C/C++, ffmpeg, Rust, WebAssembly

Please familiarize yourself with the following

Apply

Please complete the two pre-tests:

2692

2703

Add your answers under the discussion threads, or email them to me at michael AT secondstate.io

Then, you can apply here: https://mentorship.lfx.linuxfoundation.org/project/41a8bf79-a903-4c03-afb7-256fd373c0b0

Please refer to the timeline of LFX mentorship 2023 03-Sep-Nov. Since we have to process the internal review and decide, the final date for valid submissions will be Aug 27, 5:00 PM PDT.

aashi-ihsaa commented 1 year ago

hey @juntao interested in this issue under LFX. I have programming experience in c++ and rust. Will go through the sources you have mentioned and familiarize myself with them. Would be great if you could mention some additional skills that might be required to do this seamlessly.

juntao commented 1 year ago

hi @aashi-ihsaa

Thank you for your interest! Please see the updated description for two pre-tests -- they will help you evaluate the skills needed for this project. I would strongly encourage you to take those pre-tests and apply.

cheers Michael