search

Shopify / react-native-skia

High-performance React Native Graphics using Skia
https://shopify.github.io/react-native-skia
MIT License
6.91k stars 447 forks source link

Video rendering support #667

Closed laurens-lamberts closed 5 months ago

laurens-lamberts commented 2 years ago

We currently don't have a specific use case for this, but I think it would be nice for react-native-skia to support video rendering. Especially as this could provide the possibility to apply blurs on top of video content, and make for example;

Also, this could eliminate the need of adding an extra library such as react-native-video, when react-native-skia is already part of your project. This could be practical if you just want some basic video rendering.

Some related code files I found in the Skia source which might be helpful; Skia video decoder Skia video decoder ffmpeg

wcandillon commented 2 years ago

Thanks for sending us these pointers. This is indeed a hot topic. Based on the examples, I'm wondering if there is a way to access the encoded frame via ffmpeg-kit-react-native and then there is maybe not much that would need to be done on our side. By looking at the API at https://github.com/tanersener/ffmpeg-kit/blob/main/react-native/src/index.d.ts it is not clearly yet how to do it. Maybe run the command to get the jpg/png and access it via the filesystem module?

MadCoderme commented 2 years ago

I would suggest and request native android developers to look into this and use android Mediacodec API for hardware accelerated actions instead of ffmpeg.

BenCoDev commented 1 year ago

I'd love to be able to create a video timeline scrub with canvas.

I made one with react native ffmpeg but it feels pretty over-engineered. See here

That'd be a great use case for me. If I can help somehow, is be happy to!

XantreDev commented 1 year ago

I'd love to be able to create a video timeline scrub with canvas.

I made one with react native ffmpeg but it feels pretty over-engineered. See here

That'd be a great use case for me. If I can help somehow, is be happy to!

This article is crazy)

BenCoDev commented 1 year ago

I'd love to be able to create a video timeline scrub with canvas.

I made one with react native ffmpeg but it feels pretty over-engineered. See here

That'd be a great use case for me. If I can help somehow, is be happy to!

This article is crazy)

Crazy good or crazy bad? 😝

XantreDev commented 1 year ago

I think crazy good, but I've performance concerns

BenCoDev commented 1 year ago

I think crazy good, but I've performance concerns

Thanks, you're too kind. Yes, honestly, it is a bit goofy to use ffmpeg here. when I have some times, I'll do the same using canvas. It surely will be far more efficient.

hungtooc commented 1 year ago

It's been quite a while since this issue was opened, does anyone have any solution yet? @all I need to render a video from Skia animation. I just learned a little about ffmpeg-kit-react-native, it seems I have to save the entire image frame as a file and execute FFmpeg commands by providing image paths. ffmpeg-kit-test This seems suboptimal because there is only a slight change between frames. Hopefully Skia can support video rendering more effectively when the frames created by skia and reanimated3.

marcusstenbeck commented 11 months ago

I’m exploring options for overlays on videos or ideally using a video as a mask for something else. Has anyone been able to get a video surface into a format that can be consumed by RN Skia?

XantreDev commented 11 months ago

I've used animated images (webp). It looks like video, if preload all frames

marcusstenbeck commented 11 months ago

Thanks for the tip, but my use case requires video. I’m guessing makeimagefromview isn’t fast enough to run on every frame?

XantreDev commented 11 months ago

Don't know, maybe you should check it

aphillipo commented 7 months ago

I would love it if it was possible to render WebRTC video to a react-native-skia video tile as it would allow me to generate beautiful overlays that react to the underlying video - for example the designer I have came up with a graph that blurs video under it, easy to do with react-native-skia but extremely hard to do otherwise. Obviously this part of the work and a format that can be produced by WebRTC or whatever would have to be agreed on, would it just take the raw decoded frames somehow or accept an MP4/VP8 stream directly.

wcandillon commented 6 months ago

We are actively working on this now.

https://github.com/Shopify/react-native-skia/assets/306134/2e9ee537-9dae-4c0c-bf56-a6daaeef8b65

mlecoq commented 6 months ago

Can't wait to test it 😁