Understand performance of EyeCommander

[gavinhenderson]

EyeCommander can sometimes run a little bit slow. Right now we have no idea if it's hitting a memory or CPU limits. We also have no idea where the bottle neck is.

As a measurement of speed/performance we should use the frame rate that can be passed into the media pipe model. Ideally we should be hitting around 30 frames being processed a second.

Things that could be a bottle neck:

• Searching through the history of pervious frames
• Copying frames unnecessarily
• Too many canvas' (offscreen)
• React triggering re-renders

Notes:

• This model is speedy: https://github.com/emilianavt/OpenSeeFace

[gavinhenderson]

Had a dig into this, profiled the app with chrome dev tools.

Every time a frame is processed it goes through the webGL function readPixels which is slow.

I tried bypassing it by passing the image as ImageData but it still loads it onto a canvas then reads the pixels. Based on the issues on GitHub the devs seem pretty focused on performance so its likely they have the most performant setup already.

The only other option is to try out other libraries and see if any are faster. I tried out mocap4face but it was also slow on low end devices.

Will leave this open for now but going to lower its priority because improving the performance will be a lot of work

[willwade]

I wonder Gavin if we just need to do some testing across different devices/hardware and suggest a "Recommended hardware" (or maybe "Tested and works well" list"

[gavinhenderson]

So gonna take another stab at this.

First thing I am going to do is to make sure that EyeCommander is squeezing the most out of the facemesh algorithm and not adding any overhead. There is a web example of facemesh here: https://codepen.io/mediapipe/full/KKgVaPJ

I will compare the FPS from that minimal example to the FPS in eyecommander (will make eyecommander use the same FPS module). If its equal then we know the problem isn't eye commander but just limits of the machine or web platform.

I am then going to see if there is a native C++ example and see if that gets a better FPS, if it does ill think about bridging into that somehow.

[gavinhenderson]

Setup the demo here: https://acecentre.github.io/EyeCommander/

[gavinhenderson]

Good news, we get worse performance on EyeCommander, so we aren't at the limits of the web! No to do some profiling and find the culprit of the slowdown

[gavinhenderson]

Current thinking its memory thats the bottleneck, and there is maybe a memory leak somewhere

[gavinhenderson]

So current working theory is no memory leak, just memory intensive. This is mostly because its an electron application and it needs to swap every frame in an out of memory.

There is probably small memory wins but they aren't really worth it because they won't make much different.

The other option is rewrite in a more lightweight technology than electron. Is it worth it? We could just say minimum 16GB ram and be done with it? Annoyingly though that would exclude grid pads which is pretty annoying.

Im going to have a very little dig into other web based platforms and see if there is an easy migration path. My suspicion is they will all be quite memory intensive but haven't tested that theory

@willwade What do you think? Invest time now in performance (mainly for gridpads) or skip it and say 16gb min?

[gavinhenderson]

Worth noting, we are quite tied into electron specifically with our build pipeline, installer and with the auto updating feature :( So whatever we switch to would need to have those too

[willwade]

Yeah. It’s a bit of a snag. Not an easy win. Note it’s not just gridpads. Pretty much all dedicated AAC are on par - or worse - specifications. I feel the burn. I’m not sure any lighter approach will really be that significant but maybe I’m wrong?

[gavinhenderson]

Note it’s not just gridpads. Pretty much all dedicated AAC are on par

Good point, makes it a pretty big issue problem really.

I’m not sure any lighter approach will really be that significant but maybe I’m wrong?

Things like https://tauri.studio/ look promising memory footprint wise (at least according to their own marketing material), but its hard to know without really going all in. I'll do a small test to see how big the difference is really. Ill spend the morning doing some mini experiments with electron alternatives

If we ditched web tech and went fully native on windows, that would almost certainly fix our problems. But then we are talking a massive amount of work.

[gavinhenderson]

@willwade So I gave Tauri a stab and its super cool. If we ever need to build a cross platform web application we should definitely use it! It is rust based (for the backend) so would require some learning to get up to speed but thats not a huge issue.

Anyway, long story short, EyeCommander is no faster on tauri. The bottleneck is the model :(

Not sure where we go from here?

[willwade]

Well I'd say close the issue - and mark it as complete :) Job done I guess!