Shader-based smooth transitions between presets

[kblaschke]

Smooth preset transitions were broken for a long time now, and before that, they weren't really pretty. With the renderer rewrite now done, it has become way easier to blend two presets in new ways. This issue will track progress on implementing a new transition effect, which uses randomly selected shaders to blend from the old to the new preset.

These shaders will be static in the first iteration, but a later projectM versions will get new API functions to load user-defined transition shaders and possibly callbacks to select each transition on the application side. This would, for example, allow users to create preset playlists with specific transitions after each preset.

The transition shaders will have the following properties:

• Written in GLSL, so we don't need to translate them via hlslparser.
• Independent from the preset effects - they're run on top of the final image of both presets.
• Ideally cheap calculations only, as they will run on top of two full preset rendering steps.
• Support for ShaderToy syntax/variables to make it easier to test and then copy & paste the shaders for use with projectM.

The shader will receive a number of uniforms which contain different values which might be interesting to the transition. Some of these are:

• The linear transition progress from 0.0 to 1.0
• Additional progress values calculated with non-linear easing functions
• Audio analyzer data like bass/mid/treb values, as used in preset shaders
• Random values
• Possibly texture samplers with the same selection mechanism used in preset shaders

Any suggestions, wishes or ideas on this topic are highly appreciated - please leave a comment if you have some input on this!

[ehdocumentdesign]

This sounds really cool, and it relates to a lingering question I've had.

I had been using version 2.x of libprojectm on an older machine, and it seemed like the smooth transitions would progressively blend and eventually replace the old preset with the with one. This created some impressive visuals, especially when setting a long transition time. (I can upload a video clip if this description isn't enough.)

Version 4 of libprojectm is such a huge improvement on single preset rendering, but I've noticed the smooth transitions are not the same, and yes they appear broken when using long transition times.

Would the version 2 behavior, or something like it, be within the scope of what you propose here? I don't know enough C/C++ to contribute to the code, but I'd be glad to help test.

[kblaschke]

I actually never looked at the 2.x releases, but this version mostly supported Milkdrop 1.4 presets, which didn't make heavy use of fragment shaders.

Milkdrop has a very complex blending mechanism, deeply built into the rendering pipeline. Basically, if it blends two presets, each rendering step is executed twice and each effect (waveforms, shapes, motion vectors etc.) have their own, individual blending logic. On top of that, Milkdrop also applies some additional effects like a plasma effect to further add some variation into the blending.

In projectM, we opted to not copy/reimplement this process, as it only really works when blending two Milkdrop presets together which use the exact same rendering code. As we're planning to add support for other preset formats (first variants of Milkdrop, e.g. PowerAmp presets) and later on, with version 5, possibly an all-new preset format, we need a more generic blending method which isn't built deeply into the preset rendering code itself.

So what I'm proposing here is using an additional rendering step after both presets finished rendering. In this step, we draw a simple full-screen quad (AKA rectangle), which has a fragment shader that does the actual blending. That means for every pixel in the output, the shader can decide how the pixels from both inputs (textures from the old and new presets) are used and mapped/combined in the final image. The shader knows about the blending percentage, some beat values etc., so it's highly flexible.

In the end, the transitions will not exactly look like those in Milkdrop, but provide us and later the users with more flexibility, e.g. if they can add their own transition shaders and we can easily add other preset formats to the library without having to change this again.

There's one caveat for newly displayed presets: We need to start with the last output frame of the old preset and "seed" the main texture of the new preset to prevent slow-drawing presets from always starting with a black background, but this is easy to implement. We'll also do this with hard cuts to make the transition a bit nicer to look at.

[milkdrop2077]

This is excellent news 👍

[kblaschke]

Working branch is here, for those interested in following the progress: https://github.com/kblaschke/projectm/tree/feature/shader-based-smooth-transitions

Preliminary documentation, will be moved to the Wiki once the feature is finished & polished: https://gist.github.com/kblaschke/a53952cbaf4e469a0733222f09d6dc1b

[ehdocumentdesign]

Looks good so far! I had a chance to try the shader-based-smooth-transitions at 9a73a88 with a couple different settings for transition timing. Nicely done.

I think the Version 2.x behavior I attempted to describe in a previous message might be the result of the code model you've specifically said was excluded from the current implementation (and I appreciate the reasons why). No matter, though-- the current implementation looks really good in its own way. Will keep an eye on the branch for further work. Thanks!