[Feature Request] Text Effects

[zakaria-chahboun]

In the most of Linux Video Editors, We notice that the texts are handled like a picture, So we can't play dynamically with it, such as adding some animation effects to the characters of a word! For example in [Adobe Premiere Pro] we can do this easly! Also in [DAVINCI RESOLVE 15]. and all animations is keyframeble.

---

You can add some plug-ins like Mister Horse, and make this effects easy to use! You can get the IDEA by looking at this site MisterHorse-Texts

Thank You ❤️

[frink]

This is a needed feature. But I think we're still trying to figure out the best way to implement it. When do we rasterize the test? Do we rasterize to letters and then compose or do we rasterize the entire piece of text? What means should we use to warp the text in and out?

Almost any of these answers means coding ourselves in a corner. We want vector text and overlays. But it needs to be simple to create, yet intuitive to animate. I haven't seen anyone suggest a solution that allows us full flexibility yet...

[zakaria-chahboun]

i think, you can pass through the Vectors, by using SVG Animations with css and so on .. you can looking at this website: Moving Letters, it have a huge and beautiful text animations with SVG!

And maybe you can start with Qt by QSvgRenderer Class,

[Simran-B]

Don't good SVG animations require JavaScript? Couldn't QtWebkit be used to render arbitrary web contents? There are plenty animation libraries out there.

[frink]

That is probably the right way to go... Although QtWebEngine may have GPU better integrated. This seems to be the way that Qt is going at the moment. QtWebKit and QtWebEngine are similar. One is a fork of the other...

[Simran-B]

Should be QtWebEngine, I didn't know QtWebkit was deprecated several versions ago when I wrote my comment. However, I'm not sure about using a browser integration anymore - how do you control time? Is it even possible to tell the web engine "now I need the state at time x" to be able to scrub around the timeline and get a correct preview? In OBS this works fine because it runs in real time, in a linear forward fashion.

[benrob0329]

With the node pipeline being in-development, I think one way of doing this could be to add curves which can be animated.

For example, a bezier curve node which defines a vector output and that effects can be animated with (such as a wipe going down it).

[frink]

@benrob0329 are you thinking of providing the real time outputs of these curves to a browser canvas? Or what is your thought?

@Simran-B I'm not convinced that a browser embed is the right way to go either. But I do like the idea of capturing browser generated effects and adding them to a timeline. May need to record them and then manage them as video instead of vector - which would not be ideal for text which should stay vector until the last render. So while I see positives to a browser source I'm not sure that it is the silver bullet either.

However, I wonder if using SVG with animated CSS would be repeatable via curves like @benrob0329 was describing...

[benrob0329]

I don't think that we should need a QWebEngine effect just for a good titler. Seems a bit overkill IMO.

[zakaria-chahboun]

You can use other way, without pass through the QWebEngine! This is a repository on GitHub called ZCAnimatedLabel, this project is written by "Objective-C". You can figure out how they used it!

Example

[zakaria-chahboun]

Also if you are using GTK (or whatever), you can do some animations, for example:

they animate Characters of a word manually using timer, and this is a part of the code:

you can visit the blog for more informations.

[sobotka]

None of this garbage is managed.

[Simran-B]

Objective-C and GTK *shudders*

I googled a bit around to find something like a text animation library, but couldn't find something written in C/C++ for OpenGL. Found some interesting text rendering techniques however. Which approach is the best depends on how you want to animate your text. Not sure if there is even a high quality solution if you want to do something like rotate text in 3D while zooming in until a single letter fills the entire screen. The usual assumption for text rendering is that it's on a 2D surface. To cover extreme cases, you would probably need a combination of multiple rendering techniques, e.g. texture atlas for small font sizes, an improved signed distance field text rendering for medium to large font sizes and vector-based rasterization when zooming in extremely (under the assumption that there are only a few bezier curves to draw if you do this) - if you were to aim for the highest possible quality.

• Cairo is a high-level canvas drawing model for laying out pages and other views. The base elements are resolution independent paths and patterns. Cairo can translate that canvas model into GPU primitives using OpenGL (among others), but the canvas model does not often translate efficiently to the GPU.

  https://www.cairographics.org/OpenGL/

• libraries and programs that are using cairo [...] librsvg is a SVG rendering library

  https://www.cairographics.org/examples/

  I believe InkScape uses Cairo too for vector-graphics rendering. Related: #458

• [OGLFT] renders text strings as OpenGL primitives

  ... but if you want to animate it's all up to you it seems http://oglft.sourceforge.net/

• getting the best performance requires generating an “Atlas” for your font

  http://hamelot.io/visualization/opengl-text-without-any-external-libraries/

• If you generate an atlas once however, then any transformation (especially scaling up) will cause blurriness or pixelation. Signed distance field (SDF) text rendering can apparently solve this:

  Instead of storing the fonts as a bitmap, we store the distance from each texel to the edge of the character. We then use a shader to compute sharp edges.

  https://www.youtube.com/watch?v=CGZRHJvJYIg https://github.com/libgdx/libgdx/wiki/Distance-field-fonts https://steamcdn-a.akamaihd.net/apps/valve/2007/SIGGRAPH2007_AlphaTestedMagnification.pdf

• Qt Quick (in QML) uses Valve's Signed Distance Field text rendering, and is done on the GPU based on a precomputed distance field texture. [...] The SDF representation allows cheap text styling - outlines, glows and drop shadows are computed by the GPU as well, based on the same texture.

  https://stackoverflow.com/questions/44062566/how-do-hardware-accelerated-toolkits-like-qt-javafx-enable-quality-font-render Comparison of the 3 Qt rendering techniques: https://youtu.be/M4waDSXWGGw

• When the GPU is powerful enough (meaning desktop GPUs) we can even do sub-pixel anti-aliasing, it is just about adding some lines of shader code.

  https://blog.qt.io/blog/2011/07/15/text-rendering-in-the-qml-scene-graph/

• freetype-gl's implementations of signed distance fields [...] is too slow to do real time as signed distance fields for hundreds of new characters (not previously cached) are generated.

  https://www.reddit.com/r/opengl/comments/3kjz6e/real_time_generation_of_signed_distance_field/

• Signed distance field text rendering with sharp corners https://computergraphics.stackexchange.com/questions/306/sharp-corners-with-signed-distance-fields-fonts https://www.shadertoy.com/view/ltXSDB https://twitter.com/adamjsimmons/status/611677036545863680

• Better Contour Textures

  improved variant for calculating the SDF (as apposed to Valves pure-brute force method)

  http://contourtextures.wikidot.com/

• Multi-channel signed distance field generator (sharp corners, but different method) https://github.com/Chlumsky/msdfgen

• OpenGL text using one vertex buffer, one texture and FreeType https://github.com/rougier/freetype-gl (texture contours mentioned: edtaa3func)

• GLyphy is a signed-distance-field (SDF) text renderer using OpenGL ES2 shading language. [...] represents the SDF using actual vectors submitted to the GPU

  https://github.com/behdad/glyphy

• Vector texture atlas + bezier curve shader [...] [...] let the GPU render from the original vector data [...] This technique is harder on the GPU than atlas textures but avoids the need to use the CPU to render at runtime when a required glyph size is missing. It can also deliver higher image quality: atlas textures have the drawback that when text is scaled, rotated or shifted by a subpixel amount there is no longer a 1:1 correspondence between screen pixels and atlas texels.

  http://wdobbie.com/post/gpu-text-rendering-with-vector-textures/ http://wdobbie.com/post/war-and-peace-and-webgl/

• Easy Scalable Text Rendering on the GPU Possibly limited to quadratic Bézier curve segments. Glyphs must not overlap over 255 times. Supports sub-pixel rendering. https://medium.com/@evanwallace/easy-scalable-text-rendering-on-the-gpu-c3f4d782c5ac

For the web there are probably a gazillion of libraries to animate text, but as I said, it might not be possible to make a web rendering engine work in an non-linear video editor. For Canvas elements there is https://github.com/spite/ccapture.js which allows you to capture animations in your browser. With the PNG option it should actually support alpha transparency. It should then only be a matter of turning the image sequence into a video with alpha channel and import that into Olive. Don't know if there is anything comparable for other web content (JS/CSS animation of DOM elements, SVG, WebGL) but I guess not.

Maybe Skia's support for JSON animations is something to look into? https://skia.org/user/modules/skottie Skia is a 2D graphics library used in Chrome. The code to support web animation is probably part of Blink though...

[sobotka]

Cairo is complete and utter, unequivocal, garbage, and is not managed. Also, given any animation interaction needs to work within the animation framework of Olive, the idea of a poorly designed library seems like a fool’s errand.

The rendering library needs to be quite low level, and support proper RGB encodings. Glyphy might fit that bill, but it is hard to know. Also supporting CPU is mandatory for online rendering.

For the web there are probably a gazillion of libraries to animate text, but as I said, it might not be possible to make a web rendering engine work in an non-linear video editor.

This exactly. In particular, these sorts of research dives are great in that it makes it pretty clear what can’t work, and what might plausibly work. Ideally, the library would:

1. Support GPU and CPU.
2. Support arbitrary RGB encoding values in float. Specifically, not be hard coded or “managed” with sRGB assumptions.
3. Fit within animation constraints.

Etc.

[Simran-B]

Is the assumption that

• offline = realtime playback in "preview quality" = on GPU
• online = rendering at full quality for export = on CPU ?

Are exports rendered solely by the CPU, utilization of graphics units for video compression not taken into account? May playback in Olive use a CPU only or does it require at least an integrated graphics unit to work at all?

If there are only two code paths, online and offline rendering, then a fast approximation (e.g. using texture atlases) might be sufficient for playback, and for export vector rasterization could be done on CPU, per frame, for maximum quality.

Regarding RGB encoding... I tried to understand the pixel management affair, but I'm pretty much lost. What I get is that color channels can not use integer values in the range 0..255, but need to be represented as floating point values and there is a whole lot of conversion going on to deal with color correctly. Is it necessary to take color management into account if you want to draw text with sub-pixel anti-aliasing? Or is it unnecessary because color is being dealt with before and after the drawing stage?

[sobotka]

Is the assumption that offline = realtime playback in "preview quality" = on GPU? online = rendering at full quality for export = on CPU?

Mostly correct, depending on the quality of the GPU path, as per below. There is a Glossary page now to try and keep everyone on a similar terminology. It’s not “official” by any means, and the internal code might not reflect it.

Are exports rendered solely by the CPU, utilization of graphics units for video compression not taken into account?

It depends on whether or not the GPU path can achieve the same quality. Minimum quality depth is typically 32 bit float for processing. GPUs may have issues both at bit depth, and also at memory constraints. The two paths make it easy to identify where corners are permissible to cut, and where they are not.

Even a mostly inadequate offline should meet or exceed a rather low bar for quality in terms of conventional standards amongst existing open source options.

May playback in Olive use a CPU only or does it require at least an integrated graphics unit to work at all?

It is currently selectable, and hopefully remains that way. GPU drivers are notoriously crappy, buggy, and inconsistent things. At some point, folks might opt to use the CPU to avoid issues that they may be experiencing. GPUs put plenty of overhead on developers, as can be seen via the Blender blog posts on the nontrivial bugs and glitches each vendor offers up.

If there are only two code paths, online and offline rendering, then a fast approximation (e.g. using texture atlases) might be sufficient for playback, and for export vector rasterization could be done on CPU, per frame, for maximum quality.

Precisely, with caveats.

That is, if your reference space is a wide gamut, high dynamic range affair, the RGB triplets need to be rendered agnostically within the library, or at least permit the developers to control the rendering assumptions with respect to the target dynamic range / gamut. Specifically, if you think about an anti-aliased subpixel, it represents the geometry of the glyph at partial coverage. This means that the RGB needs to be calculated in the linear domain to match up with the design constraints of “reasonably close to online quality”. The same applies for RGB chromaticities. Cairo is an epic bed shit in this regard, and a generally horrible library for quality. If you want to see how horrible, use Inkscape or any other awful software that leans on it.

Regarding RGB encoding... I tried to understand the pixel management affair, but I'm pretty much lost.

Don’t be! Most of the issues come from someone’s existing understanding of pixels and management, which is almost exclusively wrong. The simplest idea to begin to grasp the concepts is to appreciate the relative nature of RGB. That is, if you are on an sRGB, Apple Display P3, or LG OLED HDR display, we know that they are different in some way. Yet no one stops to think what happens if you send identical RGB triplets to each. Even the most basic understanding of differences would reveal that sending identical RGB code values will not work.

What I get is that color channels can not use integer values in the range 0..255, but need to be represented as floating point values and there is a whole lot of conversion going on to deal with color correctly.

Depends on context, but mostly accurate. Ask yourself why you typed 0-255, for example, and you’ll begin to see why it is a meaningless format. This link might help you begin the trek of unwinding broken knowledge. It might not, too. :)

Or is it unnecessary because color is being dealt with before and after the drawing stage?

The underlying assumptions of any particular library absolutely have implications, and can’t be ignored.

[Simran-B]

Thanks, will check it out and probably read the color management discussion again.

I watched something about color management a while ago and this quote seems quite important:

Same RGB value gives different colors on different displays

https://www.youtube.com/watch?v=4QyOFiDl8Bw

For CSS4 there is something being worked on called "device independent colors": https://www.w3.org/TR/css-color-4/#lab-colors

Does this describe the first part of color management, the conversion to linear colors? https://www.w3.org/TR/css-color-4/#rgb-to-lab

[sobotka]

Does this describe the first part of color management, the conversion to linear colors?

No. That's closer to a clown car. My experience with the CSS folks is nothing short of feeling like one has driven a truck into a dumpster of stupid feel free to check out the issue 366 over at https://github.com/w3c/svgwg/issues/. I don't want to link it here as it will be tagged in the issue.

You perform management within a context, and sometimes you will enlist the capabilities of a Colour Management System. Many of these exist, such as ArgyllCMS, Apple's ColorSync, Autodesk's SynColor, etc. Different CMS approaches have different design contexts.

OpenColorIO, for example, is not suitable for all of the nuances of display referred graphic design printing etc. that ICC fills. However, neither are most ICC based CMS pipelines terribly good at animation, motion pictures, and visual effects. While there is some overlap with core concept, there is also a bit of divergence in the design choices and assumptions that make the two approaches quite different.

[frink]

GLyphy is something I've looked into before. Should really have @behdad add a bit about whether GLyphy is the right fit..

[behdad]

Doubt that GLyphy is the right fit given that it has no plans to work on the CPU.

[sobotka]

It appears OpenImageIO added text rendering. Plausible to add the shader component to upstream as an alternative code path if it doesn’t already have it.

buf = oiio.ImageBuf(oiio.ImageSpec(img.shape[1], img.shape[0], img.shape[2], oiio.FLOAT))
buf.set_pixels(buf.roi, img)

text = "test"
text_color = (25, 25, 25)
text_font = "Arial Bold"
text_size = 50
oiio.ImageBufAlgo.render_text(buf, x, y, text, text_size, text_font, text_color, alignx="center", aligny="center")

img = np.array(buf.get_pixels(oiio.FLOAT)).reshape((img.shape[0], img.shape[1], img.shape[2]))

[Simon-Laux]

Would be cool if we could animate texts properties with keyframes like size and color, currently it's only posible to have "hold" keyframes that set the whole state and not single properties. Also having svg layer for text would allow for zooming text without pixelated or blurry edges.