Add matrix options for PImage uniforms in PShader

[neilcsmith-net]

Tangentially related to work in #2963 it would be useful to be able to set a texture matrix (and offset?) from a PImage uniform on a PShader, in a similar way to how these are set up for the default texture -

https://github.com/processing/processing/blob/master/core/src/processing/opengl/PShader.java#L1348

Could be done explicitly -

PImage img;
PShader shader;
shader.set("img", img);
shader.setMatrix("imgMatrix", img);

or automagically

shader.set("img", img); // also sets up imgMatrix uniform if it exists

[codeanticode]

You can set a PImage uniform with the current API. The usage example you mentioned:

shader.set("imgMatrix", img);

is supported, all you need is to define the imgMatrix uniform in the shader code.

[neilcsmith-net]

all you need is to define the imgMatrix uniform in the shader code.

How?! How do you get access to the information to set up the correct matrix (as the link above) without accessing Texture, which AFAIK is not meant to be a part of the public-facing API?

[codeanticode]

Sorry, I overlooked the fact you were also asking about the texture matrix. Processing automatically computes a texture matrix to apply to the default texture (you can access it through the texMatrix uniform), if you want to specify your own texture matrix, you would need to compute it an pass it as a PMatrix3D or 2D:

https://processing.org/reference/PShader_set_.html

[neilcsmith-net]

I'm only asking about the texture matrix. At the moment, you can set a PImage as a uniform, but there's no way to set up the associated texture matrix without accessing Texture to do what that PShader code is doing here -

https://github.com/processing/processing/blob/master/core/src/processing/opengl/PShader.java#L1348

I believe one benefit for #2963 would be for multi-texturing, for which this would be rather useful!

[codeanticode]

Ok, I got your point. We would need to discuss the possibility of adding this feature. While I understand it could be useful in some situations, it is something that would mostly benefit advanced users, who are would probably know how to construct their texture matrix anyways.

[neilcsmith-net]

How do you get the information to construct the matrix without accessing Texture though (eg. to get invertedY value)?

It relates to a comment by @JakubValtar in #1988 too.

[codeanticode]

You can use the PGraphics.getCache(PImage) method to get the Texture object backing the PImage, in case you want to do advanced manipulations of the GL-level information.

[neilcsmith-net]

Yes, I can, but I'm not sure it's ideal for the target audience. I agree with @JakubValtar in that issue - "Especially for shader beginners, inverting is very confusing when mixing PGraphics and PImage" This is, IMO, an option for negating that problem, if making people pass through multiple texture varyings.

Incidentally, an even easier, but breaking, way to fix that would be to have all textures inverted (which would also allow for faster pixel upload).

[neilcsmith-net]

@codeanticode any further thoughts on this? Is there likely to be a fix or should I just invent a public facing API for my own purposes? It's a really useful thing missing from the current PShader API.

[codeanticode]

I agree that having Processing automatically generating the texture offset and matrix uniforms could be useful and would be consistent with the way the default texture is handled, but at the same time I would like to avoid making new additions and/or changes to the API.

The least intrusive way to implement this proposed enhancement could be to calculate the offset and matrix in set(name, img) and then pass the values through uniforms named name + "Offset" and name + "Matrix" (for example), something like:

public void set(String name, PImage img) {
  setUniformImpl(name, UniformValue.SAMPLER2D, img);
  Texture tex = currentPG.getTexture(img);

  set(name + "Offset", 1.0f / tex.width, 1.0f / tex.height);

  PMatrix3D texMat;
  // .. calculate texture matrix
  set(name + "Matrix", texMat);
}

Any other ideas?

[neilcsmith-net]

Any other ideas?

No, looks perfect! It's exactly what I had in mind by the "automagic" approach in my original issue report. As you say, this keeps it consistent with the default texture behaviour. Not sure I'd agree with it not adding to the API, just pushing it into the GLSL layer, but probably less intrusive. Thanks!

[neilcsmith-net]

OK, I've implemented myself as follows (should anyone else need it in the short term)

https://github.com/praxis-live/praxis/blob/master/praxis.video.pgl/src/net/neilcsmith/praxis/video/pgl/PGLShader.java#L46

Ideally this would probably be in consumeUniforms() but that's awkward from a subclass because the UniformValue constants are package private.

I've been using this to port GLSL.io transitions (which need some minor adaptation) - happy to contribute those upstream as a sketch example once this gets in.

Thanks!

[neilcsmith-net]

Any further thoughts on getting this in upstream? I've been doing some work porting Interactive Shader Format examples recently, but again not really usable without this extension.

Thanks!