⚠️ DEPRECATED ⚠️ Please visit the new homepage at https://grants.livepeer.org
43
stars
7
forks
source link
[Video Disruptor] Client-side AI compute : Plugin suite for Livepeer React Player and Broadcast components, enabling AI-generated virtual background, AR filters, noise cancellation, and live subtitling. #224
What is your project, and what problem does it solve?
Lightweight plugins for Player and Broadcast components enable on-device AI: virtual background, AR filters, noise cancellation, live closed captioning. They serve as an extra layer for developers using Livepeer React SDK, optimized for local execution with small models in the user's browser or phone.These are job sets that are best suited for local execution as compared to compute on node.
Please describe in more detail why this proposal is valuable for the Livepeer ecosystem
AI features like virtual backgrounds, AR filters, noise cancellation, and live captions enhance Livepeer: boosting engagement, branding, accessibility, compliance, and cost efficiency for creators and viewers.These plugins will enable platform building on Livepeer more tooling to offer users without a need to install third party software. Noise cancelling for streamers without need for high tech.
Please describe in details what your final deliverable for this project will be.
A plugin suite/collection usable with Livepeer Player/Broadcast component to enable client side free AI video compute job for generative virtual background, AR filter, Live closed captioning for VOD and livestreams and noise cancellation
How will this deliverable benefit the Livepeer ecosystem?
AI features like virtual backgrounds, AR filters, noise cancellation, and live captions enhance Livepeer: boosting engagement, branding, accessibility, compliance, and cost efficiency for creators and viewers.These plugins will enable platform building on Livepeer more tooling to offer users without a need to install third party software. Noise cancelling for streamers without need for high tech.
Please break up your development work into a clear set of milestones
Number
Description
Deliverable
Amount
Start Date
End Date
4
Documentation,tutorial and code samples to guide on how to build a video platform with Livepeer with the AI plugin suites.
Documentation and tutorials
$1,000.00
7/20/2024 UTC
7/31/2024 UTC
1
First plugin with AI virtual backgrounds and AR Filters.
This plugin will work with the Broadcast components of the react players.
Streamers can generate background and replace their original backgrounds. - AR filters ,overlay virtual elements onto the live video feed in real-time, adding visual enhancements such as animated characters, objects, or effects.
A plugin with features for Ai generated virtual backgrounds and AR filters.
$5,000.00
4/20/2024 UTC
6/1/2024 UTC
3
Third plugin with Live subtitling and closed captioning for Livestreaming and Video on demand
-The Plugin will work with Player components from the react sdk.
-Captioning and subtitling disable / enable per user request
Multi-lingual transcription
A plugin to enable subtitling with Livepeer react player.
$5,000.00
6/20/2024 UTC
7/20/2024 UTC
2
Second plugin with noise cancellation and sound effect
Streamers can enable noise cancellation feature to remove background when streaming from the browser using the Broadcast components.
Sound Effect
A plugin with noise cancellation and sound effects
$4,000.00
6/2/2024 UTC
6/14/2024 UTC
What is the total amount requested (in USD)?
15000
Specify your team's long-term plans to maintain this software and upgrade it over time
The project will be open-source and maintained by the me and the team . The core team will be responsible for fixing bugs, adding new features(AI tooling) , and upgrading the application over time.
Long term plan is to integrate as much AI solutions and plugin into the suite.
Plans are to have some bounties on Bountycaster, to incentivize devs to build more features into the suite.
Please describe (in words) your team's relevant experience, and why you think you are the right team to build this project. You can cite your team's prior experience in similar domains, doing similar dev work, individual team members' backgrounds, etc.
I am a blockchain, AI engineer and full stack web developer with over 5 years of experience , with keen interest in dapps development, AI and media tech stack. I worked on Freetyl (Joystick labs), a grantee of Livepeer video distributor grant, where I architected the current architecture for in-game streaming and streaming from browser to Livepeer .
I have been in the past, a multi prize winner in different hackathon hosted by the Encode community including the Next video build hackathon 2022/2023.
I am involved in several blockchain communities such as Reach, Bundlr , Chainlink e.t.c .Some of which are I participated as an open source contributor.
Who is your target user group? How do you plan on getting your users to use this?
Target group are developers building with Livepeer. Water cooler chat on the community discord to encourage devs to build with the suite. Full on documentation,tutorial and code samples on how to build with the suite.
How did you learn about the Livepeer Grants Program?
Livepeer discord
Was this project started at a hackathon or another web3 event? Which one?
no
Please include any additional information that you think would be useful in helping us to evaluate your proposal.
This proposal is a revised version of our previous application(https://github.com/livepeer/grants/issues/193). The application was closed because i was involved with an ongoing grant at the time which is now completed .My team and i decide to resubmit ,after months of research and testing we are clear and confident with our implementation.
The meanwhile our application was pending ,we had the chance to reach out to some web3 streaming application that agree such plugins we make their web platform stand out.
With the introduction of these plugins, Livepeer is poised to provide powerful tools for both developers and users.
As Livepeer enters the realm of AI technology, these enhancements makes so much sense.
Application by: Nescope14
What is your project, and what problem does it solve?
Lightweight plugins for Player and Broadcast components enable on-device AI: virtual background, AR filters, noise cancellation, live closed captioning. They serve as an extra layer for developers using Livepeer React SDK, optimized for local execution with small models in the user's browser or phone.These are job sets that are best suited for local execution as compared to compute on node.
Link to public GitHub repo (if applicable)
https://github.com/livepeerAiSuites
Link to demo website (if applicable)
Please describe in more detail why this proposal is valuable for the Livepeer ecosystem
AI features like virtual backgrounds, AR filters, noise cancellation, and live captions enhance Livepeer: boosting engagement, branding, accessibility, compliance, and cost efficiency for creators and viewers.These plugins will enable platform building on Livepeer more tooling to offer users without a need to install third party software. Noise cancelling for streamers without need for high tech.
Please describe in details what your final deliverable for this project will be.
How will this deliverable benefit the Livepeer ecosystem?
AI features like virtual backgrounds, AR filters, noise cancellation, and live captions enhance Livepeer: boosting engagement, branding, accessibility, compliance, and cost efficiency for creators and viewers.These plugins will enable platform building on Livepeer more tooling to offer users without a need to install third party software. Noise cancelling for streamers without need for high tech.
Please break up your development work into a clear set of milestones
Documentation,tutorial and code samples to guide on how to build a video platform with Livepeer with the AI plugin suites.
Documentation and tutorials
First plugin with AI virtual backgrounds and AR Filters.
A plugin with features for Ai generated virtual backgrounds and AR filters.
Third plugin with Live subtitling and closed captioning for Livestreaming and Video on demand
-The Plugin will work with Player components from the react sdk.
-Captioning and subtitling disable / enable per user request
A plugin to enable subtitling with Livepeer react player.
Second plugin with noise cancellation and sound effect
A plugin with noise cancellation and sound effects
What is the total amount requested (in USD)?
15000
Specify your team's long-term plans to maintain this software and upgrade it over time
The project will be open-source and maintained by the me and the team . The core team will be responsible for fixing bugs, adding new features(AI tooling) , and upgrading the application over time.
Long term plan is to integrate as much AI solutions and plugin into the suite. Plans are to have some bounties on Bountycaster, to incentivize devs to build more features into the suite.
Please describe (in words) your team's relevant experience, and why you think you are the right team to build this project. You can cite your team's prior experience in similar domains, doing similar dev work, individual team members' backgrounds, etc.
I am a blockchain, AI engineer and full stack web developer with over 5 years of experience , with keen interest in dapps development, AI and media tech stack. I worked on Freetyl (Joystick labs), a grantee of Livepeer video distributor grant, where I architected the current architecture for in-game streaming and streaming from browser to Livepeer .
I have been in the past, a multi prize winner in different hackathon hosted by the Encode community including the Next video build hackathon 2022/2023. I am involved in several blockchain communities such as Reach, Bundlr , Chainlink e.t.c .Some of which are I participated as an open source contributor.
Team members Kotai Soen : Frontend developer , web and mobile developer(https://github.com/KotaiSoen) Precious Amaechi : Ai developer(https://github.com/nixcode1) Oyale Peter : AI engineer,Full stack web and mobile developer(https://github.com/Bruksx)
Who is your target user group? How do you plan on getting your users to use this?
Target group are developers building with Livepeer. Water cooler chat on the community discord to encourage devs to build with the suite. Full on documentation,tutorial and code samples on how to build with the suite.
How did you learn about the Livepeer Grants Program?
Livepeer discord
Was this project started at a hackathon or another web3 event? Which one?
no
Please include any additional information that you think would be useful in helping us to evaluate your proposal.
This proposal is a revised version of our previous application(https://github.com/livepeer/grants/issues/193). The application was closed because i was involved with an ongoing grant at the time which is now completed .My team and i decide to resubmit ,after months of research and testing we are clear and confident with our implementation.
The meanwhile our application was pending ,we had the chance to reach out to some web3 streaming application that agree such plugins we make their web platform stand out.
With the introduction of these plugins, Livepeer is poised to provide powerful tools for both developers and users.
As Livepeer enters the realm of AI technology, these enhancements makes so much sense.