VP_LINK_FBO : A render pass for each object ?

[Daandelange]

Hey, I just noticed, there's a way to send textures (VP_LINK_TEXTURE) from object to object, but nothing for rendering on an FBO... (or did I miss something?).

Implementing some new objects (face tracker, countdown timer effect, flash effect, points renderer, etc.), I need to render several "rendered effects" on a final display output.
Currently, I guess that you have to put them in individual textures to be mixed/rendered together later, which doesn't make sense to me (regarding performance and GL logic).
Now each "render object" needs to allocate an FBO (in order to render), then extract that to pixels in a newly allocated texture (for output), which is then drawn in the fbo of the next object. Letting them render on the same FBO would be so much faster and lighter in memory usage. Also I guess there's a hardware limit on how much FBOs you allocate, different on each device, which would theoretically limit the creation of numerous render objects in Mosaic.

Would there be another way of "accumulating effects" in Mosaic ?
Is there some reason for VP_LINK_FBO links not to be already present ?

Maybe something like this ? [ fbo_object ]----fbo---[particle_effect]---fbo---[other_effect]---fbo---[window_output]
In karmaMapper I used this class as extended ofFbo : karmaFboLayer.h
This link-logic could also be of use to my surfaces proposal in #24

Maybe then there could also be something like a FboToTexture glue object.

[d3cod3]

Hey Daan, you're right, there is no rendering directly on an FBO implemented, right now there are some objects using FBO and ping pong, as the one "injecting" code inside an FBO, the lua script one, that basically is loading the lua scripting inside an FBO for rendering and passing to the outlet the texture associated, or the glsl shader one, that specifically make use of an FBO pingpong class for speed:

https://github.com/d3cod3/ofxVisualProgramming/blob/07d2ab6f348a9b74e31eaae2ef2a7bb4698ba036/src/objects/scripting/ShaderObject.h#L44

but only internally, so no cable now, but i don't see no problem adding an VP_LINK_FBO cable and start implementing new objects using it, the only thing we need to decide is what to share with this cable, maybe the ofxPingPong?

Let me know what you think ( sorry for the bad handwriting! )

[Daandelange]

Hey Emanuele, thanks for the awesome hand drawing.
My ping-pong practise is already a bit far away but it's definitely a need. (Are you sure it does like in,src3,src2,dst2,dst3,out [as in your drawing] rather then in,src2,dst2,src3,dst3,out [i'd guess] ?)
I'll look at ofxPingPong, but at a first glimpse it looks like karmaFboLayer with a slightly different swapping mechanism, so we're talking about the same stuff :) .
I also used an integrated mask FBO for generating a pixel mask to be used with shaders, probably not a need in Mosaic.
And handling sampling and fbo.numColorbuffers = 2; instead of ofFbo1 fbo1, fbo2; which might be nice for mosaic. Should we use one GLinternalformat or multiple ? (RGBA, RGB, BITMAP, GRAYSCALE would be nice to have on long term, but it might also be hard to implement and test all these formats with all fbo objects/effects)

Ideally, ofxPingPong should be setup with a single fbo allocation, dynamically allocating the 2nd one if pingpong is needed by one of the connected objects, with a neutral class interface (so that any object in the fbo chain can render with or without pingpong needs).

[d3cod3]

Hey Daan,

i'm thinking in coding a starting point for this new objects, so it would be easier to continue developing, i'm going to write the skeleton as i'm thinking it then we'll continue from there.

I'll post it here when i have it!

[Daandelange]

Ok, feel free to push to the new_video_objects branches, or separate ones. ;)

[d3cod3]

Fast question, i'm implementing the surface fbo objects, and for what i understand your idea was to have just one shared ofFbo with a incremental quantity of numColorbuffers, shared between a possible effects chain ( various fbo effect objects )

So:

• we have the fbo composer object with the main FBO
• then for each fbo effect we create some internal effect ( in an internal fbo ) and we add a colorBuffer to the main FBO with this texture reference
• then, at the end of the chain, we select one or another, for visualization, using setActiveDrawBuffer

Is that right?

[Daandelange]

If you're talking about surfaces as in #24 , they are different from fbo link-types but work in the same "chaining" way (composer-objects and link-type-chains), so surface link-types will be facilitated later on by the FBO "chaining" mechanism.

To give a mixed example, a textureRender object could accept a surface (shape) and an fbo inlets, drawing the texture to the FBO in the shape of the provided surface (circle, rectangle, polygon, etc.; maybe defaulted to the fullscreen rectangle if no shape connected ?). That would implicitly mean that any surface-enabled object needs to be able to render to a BasicSurface interface (so some conversion work).

Back to the FBO link type, as I see it, not sure if you're saying the same or not. (It's definitely not the same as your drawing above.) The numColorBuffers should not be infinitely incremental, just 2 for supporting optional pingpong objects in the fbo chain. (numColorBuffers is a moderner openframeworks way to implement pingpong, not requiring the double ofFbo buffers[2] full allocation, but basically the same as your ofxPingPong)

• Yes (and composer-object is a great name for that, feels natural)
• Not sure if I understand correctly... Each FBO effect (object with FBO in-and-outlet) renders its effect to the passed (and allocated) FBO (ofxPingPong or whatever that has a double texture buffer). On each effect, the FBO texture swaps, allowing the next one to draw the previous effect underneath the newly applied effect. But one effect can also pingpong multiple times (on itself)(ReactionDiffusionEffect for example) before the next objects in the FBO chain swap the textures.
• At the end of the chain, the FBO renders in the latest swapped state, corresponding to the end of the render chain, like any ofTexture link that can be rendered on livecoding/screen/other. (Fbo2Texture glue?)

Pingpong is probably confusing us... In fact, effects don't need to be aware of the previous effect's pixels (the swappable texture buffer), and should just blindly render to the fbo... except for ping-pong effect objects that do multiple render passes per frame, each one requiring the pixels of the previous pass.

I hope that helps to understand eachother (?) 👼

[d3cod3]

Right, i understand your description ( i think ), i'm on it then!

[d3cod3]

There it is, i've added a second outlet to the fbo effect, so there will be no need of the third and last object to do FBO2texture We have the fbo composer ( starting point ) and the chainable fbo effect ( the template for now, i'm just drawing an alpha red circle as the SUPER effect ), where at any point of the chain we can get the texture and connecting to other objects to project or whatever.

Now we have two options, write independent fbo effect objects ( fbo particles, fbo reactiondiffusion, etc.... ), or write packed fbo effect inside one object, and switching from one effect to another via the object GUI; both options are interesting, maybe it's best to adopt both, making specific effects, and effects group, for example.

I'll commit the code soon

[Daandelange]

Could you explain these 2 options in another way ?

[d3cod3]

Right, the first option is what we were talking from the beginning, fbo effect objects, each one a specific effect, easy, the second one it would be an extended version of the first, simply including into one single object a lot of effects, maybe a group of related one for coherence ( as various type of image delay, or different type of blending or whatever ), and just select one or another via the object GUI ( for example a selector with multiple lines and only one selection at the time ) Then, on the code side, it would be simply an if..else if...else if... block with all the effect variants and switching to one or another from the GUI, nothing fancy.

I don't know, it's not really important to have just the first type or both, so i suppose we'll see as we implement some effects-

I already write a simple particle system fbo effect, just to test something, and it's working all ok.

For the fbo composer and fbo effect template objects I still have to add a mechanism for re-init textures when changing resolution, but i already have one in place that is working for glsl shader and lua script, so it would be easy to do ( imagine a chain of fbo objects, inited by default with the STANDARD_TEXTURE_RESOLUTION, then finally connected to an output window for projection, then from the output window you change the resolution, at this point the final output texture will have a different resolution from the fbo effect chain, so a trigger must be sent in cascade backword, for re-init the texture at the new resolution)

FBO EFFECT-	-FBO EFFECT-	-FBO EFFECT-	-FBO EFFECT-	-OUTPUT
1280x720	1280x720	1280x720	1280x720	1280x720
1280x720	1280x720	1280x720	1280x720	<--1920x1080
1280x720	1280x720	1280x720	<--1920x1080	1920x1080
1280x720	1280x720	<--1920x1080	1920x1080	1920x1080
1280x720	<--1920x1080	1920x1080	1920x1080	1920x1080
1920x1080	1920x1080	1920x1080	1920x1080	1920x1080

With that done, they will be ready and i'll commit the changes

[Daandelange]

Ok, I think I get it. Not sure about the 2nd option too. Why would we have an object where we can switch between reactionDiffusionEffect and particleEffect (maybe this could be achieved using a subpatch?). If one needs both (rare case), you can just create the 2 objects; if needing to switch rendering between them (cf: your GUI menu?), an fboMixer might do the job, no ? To me it sounds heavy and messy to put all kinds of effects into a big "gui switcher object". On the other side, imagine a "slideShowEffect" proposing different transitions such as fade/zoom/slide/etc. with a gui. Or did I misunderstand ? (give more concrete examples, what are you trying to solve?)

It's a good idea to put some backward (opposite-flow) communication between the composer and the effects, but I think it's a waste to change sizes dynamically. I'd rather imagine a warning mechanism indicating that "ideal effect resolutions" don't match. An FBO is like a texture and should have an always identical size. Each effect rendering to it should adapt to it. If a 800x600 fbo is connected to a 1920x1080 output, either it's drawn at scale 1 not filling the whole output, or either it's upscaled with pixellisation to fill the output. There could be an fbo-inlet in the fbo-composer object to fill the FBO with another FBO of a different size.
Like:

[ fbo_composer (400x400) ]-----[particle_effect]----[other_effect]----[ window_output (any-X-any) ]
                                                                  \
                                                                   \--[ fb_composer (800x600) ]----[other_effect]----[ window_output (any-X-any) ]

[d3cod3]

not solving anything, just putting stuff on the table, so not important, we stick to the first option for now, then we'll see

i like the idea of different sizes, it would be useful to have a selector in the fbo composer GUI, with:

• noscale ( the texture will be drawed at his size )
• fill ( filling the output )

and the a button for adapt size to output, just in case

so the control will be completely in the hands of the user

[Daandelange]

Another thought about the interface and the FBO connection type, it's a bit weird thinking about these as regular connections as their logic differs a bit. It's understandable with the "pass-trough-objet" as your screenshot above (the signal is continuous from left to right), but in fact what we're doing there is : It could rather look like this :