Would like to review ShadeToy implementation approach in movie-masher

[AK-ITT]

Hi @syntropo, We have implemented raindrop/ripple effect in movie-masher(5.1.1) through react-shadertoy,

• For the implementation, we have taken react-shadertoy library mentioned above and done some code cleanup. final code files I have shared please review the attachment.

As we are planning to render the shader effect through the canvas in (5.1.1) version. for this, we have few approaches that I would like to discuss with you.

[High-level approach - First]

• Add Shadertoy effect definition in a separate file, following a similar kind of structure that is already in place for other effects.
• add custom payload for the added Shadertoy effect-definition files so that mixing-classes, effect-classes, and other required classes will know how to deal with Shadertoy code, based on the provided payload.
• as per my understanding this would be very difficult to handle.

[High-level approach - Second]

• We have observed, that 5.0.6 is more compatible with the canvas rendering operations rather than 5.1.1.
• adding a similar kind of custom implementation would be easier in 5.0.6 instead of 5.1.1

[High-level approach - Three]

• Create a similar kind of ShaderToy effect in custom SVG format and use it directly in 5.1.1.
  that you already mentioned in this thread

please share your thoughts on this.

Thanks AK

Shader.zip

[syntropo]

It's hard to comment without know if you're hoping to actually render mashes into true video format or not. If so, the third approach seems like the only possible option in Movie Masher since FFmpeg doesn't inherently understand ShaderToy or Canvas. In this case I would again point you to Remix since that effectively screen scrapes a web browser where these could be better supported.

If instead you are only trying to render the mash in a web browser the other approaches could make more sense. If you expect to continue working on the project indefinitely you might want to work with current version, understanding that some changes still need to be merged into the core library - the client changes are in the 5.1.2 branch, but not the server ones.

I'd need to investigate further, but additional changes might be possible in the current version to reintroduce canvas support for client-only deployments. The interface between React and the core library is actually agnostic about the content of each track - both SVGSVGElement and HTMLDivElement objects are supported. Theoretically at least, a canvas element could be returned.

[AK-ITT]

Thanks @syntropo, for sharing your thoughts. As per your suggestion, we are focusing on SVG-based rendering and We have found similar kinds of effects in the SVG

Sobel Operator:- https://codepen.io/mullany/pen/pgKejj Raindrop:- https://codepen.io/tmrDevelops/pen/raGBoz Mattias CRT:- https://codepen.io/alisasila/pen/qBRNJbj

We are trying to integrate SobelOperator in movie-masher. Here are the steps and high-level structure:-

1. Created 'sobeloperator.json' in moviemasher.js\src\Definitions\DefinitionObjects Folder
2. Also added the 'SobelOperator.json' in 'EffectFactory.ts'
3. Created 'SobelOperatorFilter.ts' in moviemasher.js\src\Filter\Definitions Folder
4. Added the 'SobelOperatorFilter.ts' in 'FilterFactory.ts'

I am sharing code snippets please review the attachments for reference.

Right now, we are not getting the desired output

the second image is showing that SVG classes are adding

We are not able to figure out the correct way to implement this effect Could you please review and suggest the recommended approach?

Thanks AK

SobelOperator.zip

[AK-ITT]

Hi @syntropo, Sobel-Operator effect issue is resolved. Earlier we were using some different configurations in the 'SobelOperatorFilter.ts' file, now we have changed few properties in the file.

Sharing the updated file please review it. sobel-operator-effect-working.zip

Thanks AK

[syntropo]

Fantastic! If you're not needing to render the mash into video it seems like you're all set, right?

[AK-ITT]

Hi @syntropo , earlier for us SVG-effect rendering was a hard nut to crack, but after implementing the sobel-operator-effect successfully we feel confident about the usages of movie-masher library.
Thanks for your support. Cheers!

[AK-ITT]

One thing I would like to discuss regarding 'shader-toy' implementation in movie-masher-v5.0.6

• we observed that v5.0.6 is using 'CanvasRenderingContext2D' context, in case we need to implement shader-toy codes we have to use 'WebGLRenderingContext' context. We have already created a plugin that executes shader-toy and this plugin requires 'WebGLRenderingContext' context. But we can't use 'CanvasRenderingContext2D' & 'WebGLRenderingContext' context from the same canvas.

Could you please guide us on how we should proceed from here?

Thanks
AK

[syntropo]

Can't you just create a new context manually and supply it to the class instance? I forget how flexible the class was in that old version...

[AK-ITT]

can you provide some code snippet if possible?