search

ant-media / Ant-Media-Server

Ant Media Server is a live streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency. Ant Media Server is auto-scalable and it can run on-premise or on-cloud.
https://antmedia.io
Other
4.25k stars 625 forks source link

Support beautifying effects and masks on AMS side #3936

Open phamtuan17 opened 2 years ago

phamtuan17 commented 2 years ago

Is your feature request related to a problem? Please describe. As livestreaming demands are rising, a solution to apply beautifying effects or face filters on the SDK side (both Web and Mobile app) would be a great bonus. That means the streamers can apply the beautifying effects or face filters and the viewers can see those effects on their side as well.

Describe the solution you'd like If possible, it would be great to have beautifying effects or face filters support by the server side. In case if it takes too much resource, a solution to make it works on SDK side (Web and mobile SDK) and then send the effects together with the stream contents would be nice as well.

mekya commented 2 years ago

Thank you @phamtuan17 , yes I agree with you about supporting them in the SDK side would be a great bonus.

Let me move it to the backlog to schedule

abbas-mulani commented 2 years ago

Hi @mekya,

Could you let me know how to steam FFMPEG video to AntMedia, because when I have emailed to your support team they have replied me with following text:

"So I had an internal discussion and it turns out that if ffmpeg has such a filter then it can be applicable. Otherwise we don't have such a plugin."

So, that I will search for Mobile SDK (iOS and Android) which will provide FFMPEG output and I will provide that as an input to AntMedia streaming.

Thanks.

abbas-mulani commented 2 years ago

Hi @mekya,

Could you let me know how to steam FFMPEG video to AntMedia, because when I have emailed to your support team they have replied me with following text:

"So I had an internal discussion and it turns out that if ffmpeg has such a filter then it can be applicable. Otherwise we don't have such a plugin."

So, that I will search for Mobile SDK (iOS and Android) which will provide FFMPEG output and I will provide that as an input to AntMedia streaming.

Thanks.

Any update on this ?

mekya commented 2 years ago

Hi @abbas-mulani ,

You can stream from FFmpeg to Ant Media with RTMP for instance ffmpeg -re -i SOURCE -acodec aac -vcodec libx264 -tune zerolatency -g 60 -profile baseline -f flv rtmp://ANT_MEDIA_SERVER/APP_NAME/STREAM_ID

As far as I know, FFmpeg does not have such filter out of the box. I think the proper way for this filters are using OpenGL in the client side. Here are some samples https://github.com/sangmingming/android-instagram-filter https://github.com/zhourihu5/android-gpuimage-1 https://github.com/cats-oss/android-gpuimage

I've looked for some beauty filters but I couldn't find this time. I'll update you if I find them.

peterzanetti commented 2 years ago

Is your feature request related to a problem? Please describe. As livestreaming demands are rising, a solution to apply beautifying effects or face filters on the SDK side (both Web and Mobile app) would be a great bonus. That means the streamers can apply the beautifying effects or face filters and the viewers can see those effects on their side as well.

Describe the solution you'd like If possible, it would be great to have beautifying effects or face filters support by the server side. In case if it takes too much resource, a solution to make it works on SDK side (Web and mobile SDK) and then send the effects together with the stream contents would be nice as well.

I would say that background replacement is more in demand at the moment than beautifying effects, but they are all in the same realm: these are things that should definitely be done at the server, because finding cross-browser/platform compatible ways of doing this on the client side is next to impossible. And the additional overhead it creates for the client CPU to process these effects is prohibitive. Some examples using webGL 2 perform pretty well with minimal CPU impact, but there is no universal solution for this, and having it built into the server would be a major improvement.

abbas-mulani commented 2 years ago

Hi @mekya ,

Suppose I have used third-party SDK in iOS as well as in Android to beautify video,

Then, can you please let me know how we can stream that beautified video to AntMedia using iOS and Android SDK?

Thanks.

mekya commented 2 years ago

Hi @abbas-mulani,

There is an integration work in this issue #4171 to support these effects.

In this integration work, I think iOS side is simple because iOS can receive external frames from third part SDK. On the other hand, Android SDK cannot receive external frames for now.