Image data manipulation

[jamshark70]

Currently, [gl.image] both loads and draws the image data.

This makes it impossible to do anything other than display the image directly. (Well, of course [ofelia] can handle all the requirements, but in this case, it would mean re-implementing all of the plumbing -- the whole point of the abstractions is so that the user doesn't have to rewrite the Lua code to load and manage image data.)

So it makes me think about what would be the best way to modularize.

Perhaps [gl.image path pdname] --> do stuff --> [gl.useImage pdname] --> [gl.plane]? That is, let the user decide the name under which the image data will be accessible, and then you can interpolate any number of other objects in between the image loader and the image draw-er.

Or maybe there's another way that is more idiomatic.

[jamshark70]

So for fun, I did an [ofpt2/gl.imageload]:

ofelia d $1;
local canvas = ofCanvas(this);
local args = canvas:getArgs();
M.$1Img = ofImage();
local filename, drawimage, X, Y, Z, H, W = args[1], args[2], 0, 0, 0, 100, 100;
local loaded, saved;
local previousname ="me";
;
function M.new();
ofWindow.addListener("setup", this);
if args[2] == nil then print("[gl.image] : No file");
else args[2] = filename;
M.setup();
end;
end;
;
function M.free();
ofWindow.removeListener("setup", this);
end;
;
function M.setup();
if filename ~= nil then M.open(filename) end;
end;
;
function M.allocate(list) M.$1Img:allocate(list[1], list[2], list[3]) end;
function M.clear() M.$1Img:clear() end;
function M.setimagetype(float) M.$1Img:setImageType(float) end;
function M.draw(l) drawimage =l[1] X=l[2] Y=l[3] Z=l[4] H=l[5] W=l[6] end;
function M.crop(l) M.$1Img:crop(l[1], l[2], l[3], l[4]) end;
function M.cropfrom(l) M.$1Img:cropfrom(M.$1Img, l[1], l[2], l[3], l[4]) end;
function M.drawsubsection(l) M.$1Img:drawSubsection(l[1], l[2], l[3], l[4], l[5], l[6]) end;
function M.update() M.$1Img:update() end;
function M.open(string) filename = string;
if ofWindow.exists then ofDisableArbTex() M.$1Img:clear();
loaded = M.$1Img:load(filename) end;
if loaded then print("loaded " .. filename) end;
end;
function M.save(string);
if ofWindow.exists then;
saved = M.$1Img:save(string) end if saved then print("saved " .. string) end;
end;
function M.get() return ofTable (loaded, M.$1Img:getWidth(), M.$1Img:getHeight(), M.$1Img:getImageType(), M.$1Img:getTexture(), M.$1Img:getPixels())end;
;
function M.bang();
M.$1Img:bind();
return(anything);
end;

And:

But the new hacky gain (which probably wipes out the original data but never mind for now) gives an error: "ofelia: [string "package.preload['_.x56537f8a4690.c'] = nil p..."]:3: Error in ofPixels< unsigned char >::size expected 1..1 args, got 0" ... OF documentation doesn't list any args...?

[60-hz]

So it makes me think about what would be the best way to modularize. Perhaps [gl.image path pdname] --> do stuff --> [gl.useImage pdname] --> [gl.plane]? That is, let the user decide the name under which the image data will be accessible, and then you can interpolate any number of other objects in between the image loader and the image draw-er.

Or maybe there's another way that is more idiomatic.

Yes, I agree that we need a way to modularize those abstractions.

I would like to be able to add an objet like [of.pixFX] right after image loader and without using an extra name references, so pixel access would be as easy as in Gem. Do do this, I might find a way to send the reference name of the abstraction to the FX abs: sending the unique ID reference right before the rendering bang and set it in the M.require function.

Here is a simple proof of concept of the communication mechanism in order to get the content ("myImg") of a module called M.img:

I don't know if it is the best way to do, and we might need also to create a separate object like of.texture to do the texture binding part then.

[60-hz]

About your second question, what if you replace the "." with a ":" after pixel processing [ofelia define], like: m.imgImg:setFromPixels(pixels);

[jamshark70]

Sorry for the delay. I got sidetracked with shaders (which... if I couldn't get that to work, then this other research would be pointless).

I think that would work, actually.

The workflow I have in mind is like this (haven't had a chance to flesh it out and see if it really works -- probably won't for a couple of weeks anyway):

• [of.image] outputs the ID.
• A new [of...] object for a filter:
  • Allocates an FBO according to the source image's size and type.
  • Starts a shader.
  • Sets the shader properties based on image size etc. (This is needed because the shader receives pixel coordinates, but GLSL texture2D() expects coordinates normalized to 0.0 - 1.0.)
  • image:draw() into the FBO.
  • End the shader.
  • Pass the FBO ID forward to the next drawing object.
• A geo draws it.

If the module name is going to be determined automatically by image-$0 or $0.0, it would be good to add a method e.g. getName to query it.

Another question -- why does the get output list include OF pointers? AFAICS The pointers are not useful to other Pd objects (I even crashed Pd multiple times just by trying to ignore the pointers in list operations), and, the way for Ofelia objects to access the pointers is to require the module and then thatModule.img:getTexture(). It seems risky to give users access to the raw pointers. If it were me, I would take those out.

[jamshark70]

Some good news -- after very slow initial steps, I now have a working proof of concept. Switching the FBO on and off at the beginning shows that unity gain does not modify the image contents; then the brightness control does what you expect; and the original image is still available. (And the shader is bloody fast, omg.)

https://user-images.githubusercontent.com/318301/103615456-967d0900-4f65-11eb-80bb-67c415cadbb9.mp4

This demo makes a couple of design changes.

• of.image doesn't bind() anymore. If we're going to be enclosing graphics processors within the bind/unbind pair, other than texturing onto a geo, I got nervous about nesting bind/unbind too deeply. Instead, it just passes its numeric ID forward.

• of.plane adds if(imageSource ~= nil) then imageSource.image:getTexture():bind() end; and a corresponding unbind to its bang/draw function. (imageSource is set by function M.float(f) imageSource = require("image-" .. f) end; where the float comes from the preceding image processor.)

Then the shader ofelia object has its own ID (for the demo, I hardcoded it -- that will be easy to change for an abstraction). So:

1. [of.image] outputs "its ID; bang"
2. [ofelia d] receives ID, gets image from the corresponding module table, applies the shader, and outputs "its ID [different from the image ID]; bang" -- its FBO is also called image so that downstream processors can polymorphically get the image always from that name.
3. [of.plane] receives ID, gets image from the corresponding module table, and binds it just before drawing.

That's kind of a big change from what you did, but it feels better to me...?

One thing I haven't figured out is how to pass multiple images in (for instance, to generate an alpha channel for one image based on a second image). Probably use multiple inlets... I'm sure there's a solution; I've just done enough for today.

[60-hz]

Great, an object named [of.shader] could be done, which can take a .frag and .vert as argument, that's a nice news.

of.image doesn't bind() anymore

Yes, I can understand that. Gem adds the binding inside an additional [pix_texture] object, which is a good logic after all and make all the process more clear and modular. But I like the idea to keep the number of objects numbers to minimum during workshop (less laborious), and your solution using an intelligent binding in shapes looks interesting.

So I see 2 ways: 1 - Test your solution in different situations (iteration etc..), if it works well let's go for it. 2 - Making the binding inside an [of.texture] and shifting the basic blending option from [of.draw] to [of.texture] so mimic gem process.

Then the shader ofelia object has its own ID (for the demo, I hardcoded it -- that will be easy to change for an abstraction).

I think it might good to send like "id-$0" message instead of a simple "$0" float here (and parse it in the next object later), so if a user send accidentally a float to the object it doesn't mess with its internal image reference.

One thing I haven't figured out is how to pass multiple images in

Yes, multiple inlets is a simple solution here. But using > 2 images might require dynamic patching which is a bit cranky sometimes... anyway during art workshops being able to use 2 images satisfy most of use cases.

[jamshark70]

Test your solution in different situations (iteration etc..), if it works well let's go for it.

Right. Next test for me will be iteration with a multiimage -- I'm not sure if one allocated fbo can have different contents at different times during a render cycle.

I've got a show on Sunday and I'll be focusing on that for the next few days, but I'm encouraged by this result. Also I'm very grateful for your work up to this point -- many things about OF would have taken me weeks to grasp on my own.

[jamshark70]

Finally coming back to this.

I had a doubt about shader inputs. Brightness/contrast vs levels vs frame difference etc. will all take different inputs. As far as I can see, the parameters must be passed to the shader after starting it:

shader:beginShader();
shader:setUniform1f("brightness", gainFloat);
shader:setUniform4f("color", color:vec4());
... more...
draw image...
shader:endShader();

If they all have different parameter signatures, the part between beginShader and the drawing can't be hardcoded. Then there are two choices:

• Each shader would need its own abstraction. (Easier to design; harder to maintain long-term.)
• Or, call functions dynamically for each parameter.

There is a way to do dynamic function dispatch:

M.funcs = {
set1f = function(list) shader:setUniform1f(list[1]) end;
set2f = function(list) shader:setUniform2f(list[1], list[2]) end;
};

(Maybe 2f would need to build a vec2 or simply pass the list -- I haven't gotten that far.)

Then, with a string describing the type of parameter, it can be set by M.funcs[selector](args).

AFAICS parameter values would have to be stored as floats (or as a list if you wanted to set the color by sending a message "fgcolor 255 255 255 255"). Then the sending function would have to handle conversion to vectors or other types as needed. (Or... maybe the abstraction massages the data format going into the ofelia object...)

I tend to prefer the idea of e.g. [of.shader levels ...] but it may take a bit of Lua trickery. I'm not quite there. If I can't work it out, then each shader may just need to have a separate abstraction wrapper. (I don't think it would be a good idea to tolerate an awkward parameter interface just to reduce the number of abstractions.)

[jamshark70]

Oh, got it for the parameter storage. We can have as many funcs as needed to translate the stored lists into objects acceptable for the shader.

M.funcs = {
send1f = function(name) print("send1f", name, M[name]) end;
send2f = function(name) print("send2f", name, M[name]) end;
send4f = function(name) print("send4f", name, M[name]) end;
};
M.descr = { gain = "send1f", color = "send4f" };

function M.printState();
print("Current state");
for key, value in pairs(M) do;
print(key, value);
end;
end;

function M.simulateSet();
for varname, func in pairs(M.descr) do;
M.funcs[func](varname);
end;
end;

(printState and simulateSet are only for testing.)

[jamshark70]

Ooh...

https://user-images.githubusercontent.com/318301/104566508-a6dd6400-5688-11eb-9961-78f9d2b27338.mp4

I think next is, maybe I fork this repository and apply the changes to the objects. I've got a clear idea what the data flow needs to be.

[jamshark70]

I've started on the surgery:

• https://github.com/jamshark70/Ofelia-Fast-Prototyping/tree/topic/moveToScripts -- starting to cut the Lua code out of the Pd objects and moving them into separate .lua files. (Why? 1. Editing code in the Pd interface is painful. 2. The shader script will require curly braces, which can't be entered into the Pd editor cleanly.)
  • The Lua files can't use $0. So I'm getting rid of all variables named like image$0 and changing them to M.image module variables. IMO when you have a module functioning as an OOPy object, it's not an ideal practice to dump variables into a global namespace to be differentiated by numeric suffixes.
• https://github.com/jamshark70/Ofelia-Fast-Prototyping/tree/topic/shader-support -- building on the script branch: adding imageID symbol messages coming out of the image providers, and using these in geos to require the right image module. Currently I've done only of.plane but the others should be easy. I tested the data flow from image, multiimage, movie and videoin --> of.plane, and it's all fine.
  • Because the Lua scripts don't have access to the $0 ID, the abstraction has to use Pd objects to interpolate the ID. Not difficult, just noting it.
  • Eventually basically all objects will need a method to pass imageID methods through the outlet. Not done yet.

I'm pondering shader argument defaults, and how to handle e.g. different formats for colors. 4 numbers for RGBA is easy. 3 numbers for RGB, ok, but then alpha might be... always 1, or...? I can always imagine an exceptional case.

I think this will all work though 😁

[60-hz]

Great!

Oh, got it for the parameter storage. We can have as many funcs as needed to translate the stored lists into objects acceptable for the shader.

That's a good new, I must admit that I don't fully understand how it works here as I don't have the [of.shader] object in detail and my lua skill is limited, but I trust you!

1. Editing code in the Pd interface is painful.

I did this because I liked the idea to have a single .pd file and showing during class how simple I could modify it on the fly without having to switch to an external file editor, then recreate the object or read updated script. It was working pretty well on a pedagogical purpose and with last pd version editing in object box is less painful. I use sublime when the script is becoming too heavy through. Having an object that calls a pd script that is calling an .vert and .frag file might be a bit heavy architecture isn't it? Do you plan to add a shader subfolder in the scripts folder?

I'm getting rid of all variables named like image$0 and changing them to M.image module variables

Yes, that makes sense. I don't remember why I did this but it was tied to an experimentation that I left behind anyway...

building on the script branch: adding imageID symbol messages coming out of the image providers, and using these in geos to require the right image module.

Ok, but don't forget that making a simple [of.texture] containing the texture binding function like in gem would be an easier task like I said before. And carrying the image-ID up to the end of the chain would require to make any [of] objects routing it. For example if somebody add a translate matrix betwwen the image and a shape, then the translate object need the function inside...

[jamshark70]

I did this because I liked the idea to have a single .pd file and showing during class how simple I could modify it on the fly without having to switch to an external file editor, then recreate the object or read updated script. It was working pretty well on a pedagogical purpose and with last pd version editing in object box is less painful.

I do see your point here. But, I think it would be better to create the Ofelia objects as [ofelia d -k plane-$0] -- only this text in the box -- and then click on the box to open a Pd text window.

Here's what I see when I open one of the original abstractions for editing:

The [outlet] is within the [ofelia] object's rectangle. I guess it looks great on your system, but on mine, the font is a little taller, so the large object requires more height, and Pd isn't clever enough to move the outlet down.

So, while I can agree with you about keeping the Lua code within the abstraction file, also be aware that the patch files may not look the same for everybody (but they would look the same for everybody if the [ofelia] objects are only one line, and stash the code into a text box). IMO the following would be a massive readability improvement and would require only one additional click to see the code in workshops.

I don't remember why I did this but it was tied to an experimentation that I left behind anyway

It's the Pd way of doing local names. IMO it's a weak point in graphical patchers. Object-oriented languages are much more articulate about variable scope.

Ok, but don't forget that making a simple [of.texture] containing the texture binding function like in gem would be an easier task like I said before.

I'm not sure I agree. In this case, yes, there's a trade-off between easier usage and easier maintenance. Requiring an [of.texture] makes it a little more strict for users, but reduces code duplication (easier maintenance). I think the code duplication is not severe (not any worse than the existing setup listeners etc.) and that it would be better to err on the side of usability.

And carrying the image-ID up to the end of the chain would require to make any [of] objects routing it.

I think you need this anyway. Somebody in your workshop will do [of.image] --> [of.translate] --> [of.texture] --> [of.plane] and bang, it's broken. I think if the framework can handle it, why not make it more forgiving?

In any case, it's easy:

I benchmarked this approach vs letting Ofelia forward the message. Passing "imageID xyz" in, converting arguments to Lua arrays, reconstructing the message as an ofTable and spitting it out is roughly 10 times slower than [route] here.

[60-hz]

I guess it looks great on your system, but on mine, the font is a little taller, so the large object requires more height, and Pd isn't clever enough to move the outlet down.

Yes, I am shipping my own version of vanilla puredata, with many conveniences like menus etc during workshop so students from design background don"t run away too fast ;) But ok, I feel the -k flag is a good balance between readability and portability.

Somebody in your workshop will do [of.image] --> [of.translate] --> [of.texture] --> [of.plane] and bang, it's broken. I think if the framework can handle it, why not make it more forgiving?

Yes, true. So it might be good to make a routing abstraction for easier maintenance, as it could be useful to pass other references later if the trick is working (thinking about a mesh object that could alter verticles of a prim or a shader on top of another would require also previous shader ID?).

I benchmarked this approach vs letting Ofelia forward the message. Passing "imageID xyz" in, converting arguments to Lua arrays, reconstructing the message as an ofTable and spitting it out is roughly 10 times slower than [route] here.

I didn't know it could be so heavy to do this simple task in lua!

[jamshark70]

Holy sh--... I think I did it: a working prototype.

I defined the 'gain1' shader parameter as a floating-point RGB color (which assumes alpha = 1 unless overridden) -- for other shaders, add more lines for more parameters:

gain rgbF 1 1 1

And:

Then I can define [of.image (path)] --> [of.shader gain1] --> [of.plane 500 500] and... setting gain behaves as in previous videos posted here.

Still a lot of debugging print statements to delete, and I need to shunt 'getParameter' results out to a second outlet, and I should write a few more complex shaders to test other parameter types, but this is very promising.

[jamshark70]

And... it's really easy to add basic shaders this way. Just now, I tried:

colorMatrix.vert

#version 120

void main()
{
    gl_Position = ftransform();
}

colorMatrix.frag

#version 120

uniform sampler2D tex0;
uniform vec2 dimen;
uniform vec4 red;
uniform vec4 green;
uniform vec4 blue;
uniform vec4 alpha;

void main()
{
    vec2 pos = gl_FragCoord.xy / dimen;
    vec4 color = texture2D(tex0, pos.xy);
    color.r = (color.r * red.r) + (color.g * red.g)
            + (color.b * red.b) + (color.a * red.a);
    color.g = (color.r * green.r) + (color.g * green.g)
            + (color.b * green.b) + (color.a * green.a);
    color.b = (color.r * blue.r) + (color.g * blue.g)
            + (color.b * blue.b) + (color.a * blue.a);
    color.a = (color.r * alpha.r) + (color.g * alpha.g)
            + (color.b * alpha.b) + (color.a * alpha.a);
    gl_FragColor = color;
}

colorMatrix.desc

red rgbaF 1 0 0 0
green rgbaF 0 1 0 0
blue rgbaF 0 0 1 0
alpha rgbaF 0 0 0 1

And then this worked immediately (where the brighter areas of the image are more opaque) -- that is, it took longer to make the test patch than it did to write the shader files :open_mouth:

I want to give it a few more days with the debugging statements, but I'm feeling pretty good about this.

Next steps, then, are:

• More testing (try to do a couple of shaders that are more complex).
• Clean up the code.
• Move the Lua scripts into Pd text boxes (except of-shader.lua, which requires some {} syntax and I think that will break in a Pd box).
• I'm also considering how to do a shader with two image inputs. Probably needs to be a second abstraction/script.

[jamshark70]

But ok, I feel the -k flag is a good balance between readability and portability.

It turns out (to my very great surprise) that the script files may not even be reliable!

I made a demo patch for my class:

... and found that, maybe 70% of the time, the circle simply was not drawing. But, if I reverted to your main branch, then the circle gets drawn every time.

So I git-bisected and found that the first bad commit was https://github.com/jamshark70/Ofelia-Fast-Prototyping/commit/f3c9dba48198ee396b99b6085cea71fc9ef53100 "Convert of.window to a script file" :open_mouth:

After git revert f3c9dba, then my shader-support branch also draws the circle every time.

So the most likely conclusion is that loading the of-window script in response to a [loadbang] somehow messes up the window initialization (race condition maybe?) and causes drawing to fail. That is... quite strange, but OK, that's a very good reason to get rid of the scripts. So I will definitely do that.