Critical: Please Upgrade media pipe library and use NPM library rather script

[djaffer]

can you upgrade to latest media pipe? There is some intermittent issue I keep seeing.

[djaffer]

any update @akhil-rana.

[akhil-rana]

Sorry for the delay. Been a little busy. Will look into this asap

[djaffer]

thank you!

[djaffer]

I am consistently seeing these 2 errors. By any chance do you know what could lead to it? Is the canvas input in segmentbackground not checked for null. But I am checking it should not be null.

[djaffer]

Another critical is blur not working in ios @akhil-rana not even in the latest 13 pro.

[akhil-rana]

I think this might be safari issue. I don't have ios device. Can you check for chrome in latest ios?

[djaffer]

Checked on ios not working in chrome too.

Checked your implementation looks you are using canvas filter. Very simple fix just adds a polyfill. We need this ASAP before resorting to make the private fix.

https://stackoverflow.com/questions/51549143/context-filter-not-working-on-safari https://github.com/davidenke/context-filter-polyfill

Please see other implementations

These 2 work fine.

https://volcomix.github.io/virtual-background/ https://vinooniv.github.io/video-bg-blur/

[akhil-rana]

I suspected this too. Canvas filter should be supported on chrome though. But not sure how chrome on ios handles this. Works alright on macOS (even in safari)

Anyway I'll add the polyfill as a fallback.

[djaffer]

Thank you so much! The other parts are working flawlessly but need better error handling. Maybe put try-catch something when you can. Why does it say width is 0.

Why do you want to limit it to chrome only if it is working in safari and blur is only giving an issue?

[akhil-rana]

Chromium is a standard now a days and safari usually follows it later on. So focusing on chrome first usually solves everything else.

But yeah it's a major hassle. I've tried other methods for canvas blur but nothing is as good on performance as canvas filter. But fallback should be there anyway I think.

Pretty sure safari will add it down the line as they always do.

[djaffer]

Yes the safari is slow not sure when they will natively add but huge population who use ios use it so cannot let it be ignored from our side.

[akhil-rana]

I am consistently seeing these 2 errors. By any chance do you know what could lead to it? Is the canvas input in segmentbackground not checked for null. But I am checking it should not be null.

I can't reproduce this issue on my own. Atleast not in my demo project https://demo.virtualbg.akhilrana.com/

[akhil-rana]

Checked on ios not working in chrome too.

Checked your implementation looks you are using canvas filter. Very simple fix just adds a polyfill. We need this ASAP before resorting to make the private fix.

https://stackoverflow.com/questions/51549143/context-filter-not-working-on-safari https://github.com/davidenke/context-filter-polyfill

Please see other implementations

These 2 work fine.

https://volcomix.github.io/virtual-background/ https://vinooniv.github.io/video-bg-blur/

Hey @djaffer
I looked into the code for https://volcomix.github.io/virtual-background/ here as you suggested that it is working for the blur feature. And it looks like that its also using ctx filter as described here in the readme and used here but it still works as you say. So I'm a bit confused and I think apple might have added ctx.filter to safari (As I've tested on latest macOS)

Though I'll still work on adding a fallback just in case.

[djaffer]

Hi Macos and ios Safari might not be same in terms of feature support. You can make an account on https://www.lambdatest.com/ and see if it works or any other simulator.

[djaffer]

@akhil-rana any update? Can you just upgrade the version very least.

[akhil-rana]

Hi there is no update available. I've compared the cdn files from my code vs the latest. They are the same.

[djaffer]

Ok. When I checked there was.

[djaffer]

The mobile performance after trying multiple times seems to have an issue. Not very stable.

[akhil-rana]

I think that's the issue with mediapipe itself.

But I don't think they have made it with mobile devices in mind. It relies on all the new technologies like wasm, simd, and latest webgl. Many devices lack software or hardware requirements for these.

Also I've not seen google itself promoting background effects on mobile devices. for eg. Google meet and all.

I would say yes you can technically do it for mobiles for many browser. But a native solution in an app will perform highly better.

Also screen background is not supported on mobile devices because of the Screen Capture API which is not available on any mobile device afaik

[djaffer]

It is working fine on mobile but not reliable and sees errors. That is ok if you do not want to cater for mobile. Thanks for spending time on this.

It is supported for mobile. The official selfie segmentation demo works flawlessly in mobile despite what it says. https://codepen.io/mediapipe/pen/wvJyQpq

The only issue I can recognize is blur canvas in iOS safari other than that it works well.

[akhil-rana]

The canvas filter property is the best way to blur the canvas as I've found.

Other workarounds like forcing Gaussian blur and all have a major performance hit and specially on mobile devices will a little too much. So I'm not sure if I should do it.

[djaffer]

Yeah, most likely we will migrate and write our own due to the issues we are seeing and errors in the console not knowing what the issue is.

Would it be possible to put a demo of your minimal code in code sandbox. I have opened bug for the media pipe but they need to repro. @akhil-rana

[akhil-rana]

I also couldn't reproduce it in my demo as you can see https://demo.virtualbg.akhilrana.com/

I also have the source code for this on github. Maybe you can check it out. It's not made in react though.

Though it shouldn't be very hard to create a sandbox with basic code snippets from the docs. I may do it someday.

[djaffer]

your website does not work even on cell phone

[akhil-rana]

I've just added the alert just in case. There maybe some UI related issues that can be fixed by using desktop mode in landscape orientation maybe if you really want to use it..

I've mainly added the alert because of performance issues on low end devices and Screenshare not being supported by any browser till date on mobile

[djaffer]

The website on android high end phone does not load the camera. Checked multiple phones. You have wrapped in try catch so no errors showing. This is pretty basic.

[akhil-rana]

This is pretty weird.

I just tested on 2 devices (android) on chrome. And camera permission pops up and it works for me.

Can you test on any other browser too? Maybe brave or opera. Works for me in these 2 also.