Adding additional sampler2D uniform support

[rectalogic]

Attempting to add a new sampler2D uniform to the template transition freezes the editor (in Chrome 37.0.2008.2 dev)

uniform sampler2D luma;

I have a wipe transition that can be configured with dozens of unique luma textures to create unique transitions, if glsl.io supported additional texture uniforms this could be ported.

See: https://github.com/rectalogic/rendermix-basic-effects/tree/master/assets/com/rendermix/Wipe

https://github.com/rectalogic/rendermix-basic-effects/blob/master/assets/com/rendermix/Wipe/Wipe.frag

[gre]

I need to investigate on this freeze bug (there is an infinite recursion somewhere..) Adding the sampler2D support in custom uniforms is interesting, I'm wondering how the user texture would be stored in the gist (maybe using base64). Because all the defaults are stored in a Gist https://gist.github.com/gre/b6720916aa3f035949bc#file-uniforms-default-json in order that a gist transition can be run out of the box (with no parameter, just the defaults).

Do you have example of what a "luma" texture looks like?

[gre]

What is rendermix BTW? One of the goal of GLSL.io is to provide transitions for Video Editors :-)

[rectalogic]

You can see sample luma images here https://github.com/rectalogic/rendermix-basic-effects/tree/master/assets/com/rendermix/Wipe/lumas Sample renders of the Wipe glsl transition using those lumas are on youtube here, see the videos named "transition" e.g. this one uses the clock luma.

You can store images directly in a gist, but I would suggest just storing an image URL. You can then just create an Image element in JS and set the image src and crossOrigin attributes, and set the shader uniform to the image - as long as the image host supports CORS that should work.

The goal of rendermix was to implement a video mixer using GLSL shaders to implement transitions/effects. This is also what it's precursor WebVfx does - WebVfx allows you to write transitions/filters/effects using WebGL and use those in a video processing pipeline as an MLT plugin. WebVfx is based on QtWebKit, but you need a version built to enable WebGL. QtWebKit has been abandoned and the replacement QWebEngine will not have the features WebVfx requires.

So I wrote v8-webgl - a partial implementation of WebGL using desktop OpenGL and the v8 JavaScript engine, along with ANGLE to translate the WebGL glsl shaders to desktop OpenGL - independent of a web browser and designed to be used as a video editor plugin. Then I wrote rendermix (based on the jMonkeyEngine videogame engine), and some others as attempted replacements.

All that said, I think the most straightforward way to make glsl.io work with video editors is to write a frei0r plugin. ffmpeg, mlt and others all already support frei0r. I think the plugin would need to use ANGLE to translate the WebGL GLSL transition shader to OpenGL GLSL.

So basically a frei0r mixer2 or mixer3 plugin that would initialize an OpenGL context, convert the transition glsl from WebGL to OpenGL using ANGLE and compile the shader program, then in f0r_update2 upload the image bits for each frame with glTexImage2D, run the shader program, download the resulting bits using glReadPixels and set those in the f0r_update2 outframe argument.

[gre]

It seems that most of these luma can be made programatically but I like the concept. I like that with one luma you can easily implement all those "crop" kind of transition (I mean by crop, that it is just a matter of displaying either "from" or "to"), but also I see that it can be super powerful mixed with other effects and is a way to "compose" them.

My issue with "how to define the default image in the gist" was more that I like to keep the gist standalone (using an URL bring a dependency). The reason that a transition have default uniforms is that it can at least work with "recommended" values (and eventually it will have multiple presets in the future) and without extra custom values per transition (so easy to generalize for making a slideshow with 100 automatic effects for instance), even if I think the final software should better expose the user to change the parameters (for a better customization).

To go back on sampler2D uniforms, the solution of embedding the image in base64 is definitely ugly, URL may be ok. But I'm wondering, does that make sense to provide a "default" texture? Maybe we could make an exception that there is simply wouldn't be any "default texture" and that it should remain extern / provided by the software or the final user. A transition which requires custom sampler2D are just a bit more special. And for instance, GLSL.io could provide default textures that you can easily switch between (like in https://www.shadertoy.com/ )

PS: AFAIK you don't always need the distant host to support CORS but you can just use img.crossOrigin = "anonymous" which make the request without the cookie (but maybe I'm wrong).

About the other discussion, I'm very interested and I opened another issue #56 where I quoted you and I'm going to ask some questions :-)

[gre]

FYI, this issue is now about adding the sampler2D support, so for the short term fix I opened #57

[rectalogic]

You are correct many of these lumas can be generated programmatically, quite a few of the ones I linked to were - see https://github.com/mltframework/mlt/tree/master/src/modules/lumas

So one approach might be to port the luma generation code to GLSL as part of the shader itself - bit of a pain though.

As far as a default texture, if you don't provide one then texture2D(luma, p) will just return black (vec4(0.0)) - so that seems like a reasonable default.

[gre]

Hi @rectalogic,

I've added a basic working version of sampler2D texture support in uniforms as you asked :-)

I've made (with GIMP) a few "lumas" to work with (listed here), my idea in this first sprint is, for now, to not have too much sample textures (but we could imagine more textures, and not only for luma purpose).

It would be great to use URLs but I let this possibility for the future because I need to think about the implication of that (e.g. in term of security).

---

It is not published on glsl.io yet because planned for next release but there is already a pretty robust base.

However, You can test it here: http://staging.glsl.io/transition/new

with for instance this code:

// This template should ONLY be used for DEVELOPMENT
#ifdef GL_ES
precision highp float;
#endif
uniform sampler2D from, to;
uniform float progress;
uniform vec2 resolution;

uniform sampler2D s;

void main() {
  vec2 p = gl_FragCoord.xy / resolution.xy;
  gl_FragColor = mix(
    texture2D(from, p),
    texture2D(to, p),
    step(texture2D(s, p).x, progress)
  );
}

You can use "Save" but please do not "Publish" yet because it won't work on GLSL.io.

I'm looking forward for hearing your feedbacks, and if there is bugs, feel free to tell me.

Cheers

[rectalogic]

Nice! Here is a wipe transition, saved but not published http://staging.glsl.io/transition/00973cee8e0353c73305

Something seems to be wrong with the float uniform editor - I can't type a "." in it, e.g. if I want to enter 0.252

[gre]

very cool!

Yeah there is an issue with the number input since I moved to React, I need to fix that ASAP #71

[gre]

The basic feature is available. Closing the issue.

I've considered using imgur for supporting user images. This will come later ( #72 )

[gre]

BTW could you publish your luma transition now? v1.4 is released on GLSL.io :-)

[rectalogic]

Hmm, I get "Save failed" when publishing https://glsl.io/transition/00973cee8e0353c73305

console has "Uncaught TypeError: Converting circular structure to JSON"

[gre]

ok i'm on it

[gre]

it should be fixed (you may need Shift+R to force refresh)

[rectalogic]

Thanks, published.