search

nexmo-se / video-api-effects

MIT License
4 stars 8 forks source link

Add a no-op background effect to ease initial downloading if no effect is used? #12

Closed GMacAussie closed 2 years ago

GMacAussie commented 2 years ago

We are incorporating this library into our product (thank you!), however it seems there are a couple of choices with the background effect management:

  1. Don't wire in the BackgroundEffectProcessor output stream initially, and re-create the publisher (with the new video stream) if the user switches on the background, thereby the assets are only loaded when the background is switched on. This leads to an unpublish/publish which is not a great experience; or
  2. Wire in a blur background effect, but switch it off if the user has not enabled a background effect. This causes the assets to be loaded which is not great but not terrible either; or
  3. Am I missing something so an effect can be wired in initially which doesn't do anything and doesn't load the assets?

Given (3) is not applicable, is it possible to have a No-Op/Null/Pass-Through background effect that would do nothing - including not loading assets - which can then be loaded over later on if the user selects an actual background effect?

GMacAussie commented 2 years ago

This is a bit more vexing than we initially thought. In essence, we are trying to make sure that having no initial/switching off background effects does not incur the CPU penalty (as mentioned in the pause API).

We have come to the conclusion that the only way to achieve this is by killing the existing publisher and recreating with the required videoSource (being the default camera or background effect output). This turns out to be a bit of an issue for us as we are using the AngularJS OpenTok wrapper (https://github.com/aullman/OpenTok-Angular), and we see some issues with the DOM element being destroyed - we are not sure why preventDefault() is failing to stop that but we are continuing to investigate.

Also, note that using setVideoSource() against the publisher won't work when trying to set the background effect processor output source since it is marked as 'custom' which the OpenTok library rejects (I believe it is the 1400 error).

To summarize:

  1. Going from no effect to background effect, recreate the publisher with the new video source;
  2. Going from background effect to no effect, recreate the publisher with the new video source OR possibly use setVideoSource() with the original camera (and destroy the background effect processor). Given you still need to do (1) the video source switch is probably added work.

Suggestions welcome.

GMacAussie commented 2 years ago

Sorry for the all text but I figure I might as well finish this off. We implemented the publisher w/ DOM create/destroy by using AngularJS ng-if. I have tempered my original question re a No-Op/Null/Pass-Through background effect however with some observations:

  1. This library is pretty general, and can be used for any video source for any purpose, not just OpenTok. As such, it has no clue about 'publishers' a la OpenTok;

  2. Given (1), it is the implementors responsibility to 'glue' this library into use with the OpenTok publisher, being input stream, output stream and the publisher object. This means that the implementor is responsible for set up/tear down of the effect;

  3. To be able to have a No-Op/Null/Pass-Through background effect would require that this library is publisher-aware. That is, it would take care of getting and setting the publisher video source. It would also require some tinkering with the OpenTok publisher implementation to allow setVideoSource() to work - it won't currently as it expects an actual camera not a 'custom' video track. It raises a 1400/SET_VIDEO_SOURCE_FAILURE in that case. The only other way to change publisher video source is to perform initPublisher with the changed videoSource setting which is a pretty big hammer with implications.