UV offset / repeat should be part of materials rather than textures

[QuaziKb]

UV offset and repeat (and probably some of the other texture properties) are uniforms which are strangely passed from the texture into the material, which leads to issues where you have to clone textures per material (wasting a ton of memory) if you want per-material offsets/repeats, or roll your own custom shader. It's very inefficient to force people to do this, the best case being if someone wants to tile a texture across a surface based on the size of the object it's applied to, or use a texture as a spritesheet for animations. I can only see the current system as a hindrance with no real advantages, since the likelihood of needing shared UV offsets/Repeats on a "shared" texture is low, and would normally be better served by sharing a material. In anycase updating the uniforms off the texture is weird, since it really has no business being part of the texture, and ends up being confusing to the end user. For example, changing the offset/repeat when more than one map is applied to the material, example diffuse+normalmap, only uses the offset/repeat of the diffuse map, so the value of the normalmaps offset/repeat is completely useless in that context, when it really suggests it should affect the normal map offset/repeat. It's all around confusing.

I'm pretty sure the change is required in the "refreshUniformsCommon" function, but there's probably more to it. There's some changes required in the sprite plugin aswell. This would probably break a lot of peoples projects but it's a pretty big inconsistency in the texture/material API. It might be a better idea to make it an override that's normally null for materials, and when set ignores the values in the textures, just so it doesn't break everyones stuff.

[WestLangley]

Related post: https://github.com/mrdoob/three.js/issues/3549

you have to clone textures per material (wasting a ton of memory)

In what sense are tons of memory wasted?

the likelihood of needing shared UV offsets/Repeats on a "shared" texture is low

What do you mean by that?

If we keep the current approach, one thing that needs to be changed, is when a texture is cloned, the shared imaged should only be passed to the GPU once.

[QuaziKb]

if you have an object with a large number of frames/sheets, there's many useless objects that are created, and as you stated, the Image is copied multiple times onto the GPU which is extremely wasteful. Even more wasteful, if you have multiple objects which need to be at different points in different animations, you'll have to create a unique set of textures per unique material/object, so it quickly becomes annoying to handle, where a material uniform could much more easily be manipulated without having these extra texture offset/repeat data-structures built around a badly thought out API just to get it working. I have major slowdown in a certain use case of my application where i'm having to create these unique texture groups everywhere, and feel like the only solution would be to replace all the stock materials i use with shadermaterials to work around this single issue, or else fork my THREE.js version and make the necessary modifications.

In regards to the second statement:

I meant that the current paradigm only facilitates work in cases where you want to apply the same texture with the same UV offset/repeat, but thats generally a rare case, since it seems much more likely that you'd want to define this like other uniforms, per material, and share a material rather than just the texture.

The main caveat with the current sys. is that the offset/repeat only really exists as a single uniform which affects all textures in the shader, yet the fact it's attached to THREE.Texture means it can't be used properly as a uniform and fools people into thinking that offsets/repeats can be chosen differently for the various textures which can be applied on stock materials (i.e. you can define a different diffuse offset and normal map offset, but this isn't actually possible even though they can be set uniquely on the different textures). I can see it might have been remnant from the canvas renderer but it just makes no sense with the webGLrender / GLSL material system, which has largely usurped it.

mrdoob stated that it would need to be of the following form as a reason against having it per material,

material.map material.mapOffset material.mapRepeat material.env material.envOffset material.envRepeat ... etc.

but again, the offset / repeat don't work per-map-type even now, so this isn't really a valid argument. It would waste too many uniforms anyway to have it per-map-type, (not to mention you usually only use one set of UV's so you wouldn't really have multiple offsets for a normalmap / bumpmap / diffusemap / specularmap). I think if you weigh all the options reasonable, it really should be a material property rather than texture property. I really don't feel there are any advantages to the current system. I feel animating materials through the UV offset is a major reason to even have the property at all, and can't even properly used for that. Not to mention if you didn't need those uniforms they could be omitted from the shader compilation, whereas now they're there as long as a map is.

[WestLangley]

@QuaziKb With all due respect, if you are going to propose a fundamental change to the library, such as the one you have proposed here, you have to provide a clear and compelling argument for that change. Arguing from the frame of reference of your particular application is not going to cut it.

That being said, I too, would like to see offset/repeat moved from Texture to the materials.

I believe it is reasonable to assume the following maps have the same offset/repeat values:

map
specularMap
normalMap
bumpMap
alphaMap
aoMap (future)
glossMap (future)
displacementMap (future)

The single exception is

lightMap

This is the way it is implemented now; even though each map has its own offset/repeat values, they are not honored.

So, we could add to MeshPhongMaterial, MeshLambertMaterial, MeshBasicMaterial, and SpriteMaterial, we would add

mapOffset // or mapTranslate
mapRepeat // or mapScale

We would remove offset and repeat from Texture.

In this way, implementing a sprite sheet is straight-forward. Each sprite has a material, and those materials share a single texture. The texture has an image. The texture no longer has to be cloned. On the GPU side, the materials would share a single shader program.

IMHO, this is a more-natural API.

I am trying to recall why the API was implemented the way it was. I expect there was a good reason.

[QuaziKb]

Sorry didn't mean to suggest it was particular to my example, simply that the current api leads to significant bloat when used for animating sprite sheets / offsets per unique object/material, with no real solution. Naturally the main purpose of having this as a uniform rather than simply baking the offset/repeat into the vertex UVs of the model is to animate the offset, and i was suggesting this can't be done without going around the API, making the uniform much less useful attached to textures than if it were to the material.

[jcarpenter]

I believe it is reasonable to assume the following maps have the same offset/repeat values: map ... alphaMap

FWIW, this behaviour (the current implementation) is causing significant frustration for me at this very moment. I am attempting to animate the reveal of a mesh by tweening the mesh material's alphaMap offset independently from it's diffuse map (which stays fixed). After some frustration I discovered via https://github.com/mrdoob/three.js/pull/4987 and this bug that this is not possible.

[WestLangley]

@jcarpenter What is the "bug" you are referring to? What is your suggestion for improvement?

[jcarpenter]

Correction: by "bug" I meant this issue. A mixup due to excessive time in a Bugzilla culture. :p I do understand that this is not a bug, but rather the intended behavior.

WRT to an improvement, based on my experience with traditional 3D content creation apps like Cinema 4D, I imagine the user being able to both:

• define offset/scale/repeat for each texture within a material, independent of the other textures, and
• define the parent material's offset/scale/repeat

Alternatively for the use case I'm working on ("attempting to animate the reveal of a mesh"), it would be fantastic to have an option for wrapS and wrapT that does not wrap at all. Per the attached GIF from Cinema4D, which has an option to disable tiling entirely for UVW mapping. Based on fairly extensive testing with the existing wrapS and wrapT methods, nothing like this is possible. Meaning my options for animate the reveal of items in three.js seems limited to tweening position and opacity of the entire mesh... Unless I'm missing something.

[mrdoob]

Agreed. The plan is to simplificate all this (using a single matrix3 in the shader) and having a offset/repeat (or translate/scale) per texture.

Anyone that wants to help with this would be much appreciated!

[QuaziKb]

@jcarpenter

sadly the only way to do this right now is with multiple meshes w/ different materials, or a custom shadermaterial. Both of which are kind of involved for a user who's just jumping into three.js workflow.

There's a constant tradeoff between usability, performance and extensibility. My suggestion would be to rewrite the way textures and materials work at present in the api, so that textures are strictly parameters you plug in, and the values that are actually uniforms in the shader like offset/repeat/wrap mode are linked specifically to the material. In some cases you don't want a uniform wasted on controlling UVs for textures that never need them to change (it'd be a huge waste to have 4*5 extra uniforms in a phong material if you only need it for diffuse), so it'd be cool if there was some magic behind the scenes that detected if specific UV's are needed and the texture is rebuilt to meet those demands, or if some parameter could optionally be passed in to specify the number of required adjustable UV's & what maps they offset & how, but it's a difficult issue to resolve.

[WestLangley]

The plan is to simplify all this (using a single matrix3 in the shader) and having a offset/repeat (or translate/scale) per texture.

@mrdoob

1. Do you mean per material.map so the texture transform is in the material? There are a lot of material maps, unfortunately. We could continue to assume all texture transforms are the same, except for lightmap.
2. Do you want to support rotation, too? If so, you have to also add the rotation center -- unless you want to hardwire the rotation center to the center of the texture.

[titansoftime]

This would be a much appreciated change. Having to clone textures just to put them at unique repeat values feels awfully cumbersome. Is this how other libraries do it?

[mrdoob]

1. Do you mean per material.map so the texture transform is in the material? There are a lot of material maps, unfortunately. We could continue to assume all texture transforms are the same, except for lightmap.

I think we should have a mat3 per map and should be composed from the Texture properties.

1. Do you want to support rotation, too? If so, you have to also add the rotation center -- unless you want to hardwire the rotation center to the center of the texture.

Rotation yes. Center or not... I'm not sure, but as far as I understand, all this can be encoded in a single mat3 for the shader (per map).

[WestLangley]

Here is a prototype showing how a Matrix3 can be passed to a shader and represent a transform defined by offsetX, offsetY, repeatX, repeatY, rotation, rotationCenterX, and rotationCenterY.

If center is not allowed as an option, then it should be hardwired to ( 0.5, 0.5 ).

Comments welcome.

EDIT: demo updated

[mrdoob]

THIS IS GREAT! :+1: :+1:

I think I would go with translation and scale (instead of offset and repeat) though.

[WestLangley]

What, exactly, should the new material properties be? There are a lot of material maps.

What should be removed from the texture properties?

I am assuming that wrapS/T should remain on the texture.

[mrdoob]

I think texture.offset and texture.repeat should be removed.

I think the new properties should be... texture.translation, texture.rotation, texture.scale, texture.center and texture.matrix. We will probably also need a texture.updateMatrix() method which would be called at render time. This, of course, has the challenge of making sure we only do it once even if the texture is reused.

[WestLangley]

Referring to the title of this post, and to the arguments in https://github.com/mrdoob/three.js/issues/5876#issuecomment-69483293, I thought the point was to move these properties to the material.

[WestLangley]

/ping @bhouston for comments.

[bhouston]

My thoughts are that the texture offset/repeat should be baked into the UVs as much as possible. It is easier. This is how UE4/Unity 5 do it for the most part.

The exception is that Unity 5 does have the ability to specify one global offset/repeat per material that is shared across all textures though -- but it doesn't affect what are considered to be baked maps such as the lightMap or ambientOcclusion maps (those do not make sense to be adjusted.)

The reason why I do not advocate for a lot of flexibility here is that professionally created models have proper UVs that make this generally not necessary -- content creation tools have ways of baking this. The other issue is that WebGL has a ridiculously low lower limit for Fragment Uniforms -- 16 Vec4. If you have to have a repeat/offset per map, and we will get a lot of maps soon, we are wasting Fragment Uniforms for very little value.

I actually added a repeat/offset and then brightness/gain controls per texture in Clara.io recently and I will be undoing these changes because it is leading to overflowing the Fragment Uniforms on low end devices -- such as every Apple iOS device. Although having repeat/offset and brightness/gain works great on desktops with NVIDIA 980s, but we have to design for everyone, not the high end machines. ;)

We have to treat Fragment Uniforms with respect and only use them when necessary. This I believe isn't one of those cases.

[WestLangley]

The exception is that Unity 5 does have the ability to specify one global offset/repeat per material

This is basically what three.js is doing now -- although it gets the offset/repeat from the diffuse map, and all other material textures use the setting from that map.

The proposal is to remove the offset/repeat from the texture, and add it to the material instead -- with perhaps a name change. Again, all the material textures would share the same settings (even though some users would not be happy about it). This would keep uniform usage low.

Unfortunately, we are not getting a lot of consensus on this.

[bhouston]

Doing what Unity 5 is doing is a good idea. I would put it on the material instead of the texture as well. You have my support.

[QuaziKb]

Per map isn't really ideal, but it could be solved by having it specified with flags/keys somehow when the materials built, in case it is needed.

[bhouston]

Generally the only reason to modify UVs in the shader is because you want them animated. If you just want to transform the UVs statically, we shouldn't be modifying the shader to add this functionality.

[WestLangley]

The only proposed change to the shader is to replace

vUv = uv * offsetRepeat.zw + offsetRepeat.xy;

with

vUv = ( uvTransform * vec3( uv, 1 ) ).xy;

[mrdoob]

Ok. How about this... We add texture.dynamic (false by default) which produces:

vUV = uv;

If the user sets texture.dynamic to true then we compute texture.matrix out of texture.translation, texture.rotation, texture.scale and texture.center, we pass that to the programa and we produce:

vUv = ( uvTransform * vec3( uv, 1 ) ).xy;

Of course, if texture.dynamic changes we need to recompile the program.

That way we get the best of both worlds?

[WestLangley]

@mrdoob

1. In your view, is scale a property of the texture or is texture.scale a property of the material? I hope it is the latter, because that is what this thread is all about.
2. We do not need a .matrix property. The new Matrix3 is a uniform that replaces the material uniform offsetRepeat. It is computed from the other parameters in the renderer and passed to the GPU just like any other uniform.

[mrdoob]

1. In your view, is scale a property of the texture or is texture.scale a property of the material? I hope it is the latter, because that is what this thread is all about.

I disagree with this thread. I don't think we should pollute the materials with mapMatrix, envMapMatrix, ... Nor mapTranslation, mapRotation, mapScale. I think it's cleaner if THREE.Texture has translation, rotation, scale and, maybe center.

1. We do not need a .matrix property. The new Matrix3 is a uniform that replaces the material uniform offsetRepeat. It is computed from the other parameters in the renderer and passed to the GPU just like any other uniform.

We kind of need something like that. Say that one reuses the same texture in different maps. We don't want to be computing the Matrix3 for every instance.

[titansoftime]

I disagree with this thread. I don't think we should pollute the materials with mapMatrix, envMapMatrix, ... Nor mapTranslation, mapRotation, mapScale. I think it's cleaner if THREE.Texture has translation, rotation, scale and, maybe center.

Would a texture still need to be cloned to have different repeat properties? In small scenes this is probably no big deal, but in large ones where there already are 40+ textures this a memory nightmare.

[mrdoob]

@titansoftime image should be decoupled from THREE.Texture. Ideally one single THREE.Image could be used by different THREE.Texture each one with different translation, rotation, ... configurations.

[titansoftime]

@titansoftime image should be decoupled from THREE.Texture. Ideally one single THREE.Image could be used by different THREE.Texture each one with different translation, rotation, ... configurations.

That makes sense. Does texture.clone() already work this way, or does it have to be set up manually?

[mrdoob]

texture.clone() works that way, but WebGLRenderer is unable to know that the image is the same so that's where the texture upload is unnecessary...

[WestLangley]

I disagree with this thread. I don't think we should pollute the materials with mapMatrix, envMapMatrix, ... Nor mapTranslation, mapRotation, mapScale. I think it's cleaner if THREE.Texture has translation, rotation, scale and, maybe center.

This is basically what we do now. Each texture has its own offset property, and the individual offsets are not honored -- the renderer uses the same offset for each map of the material.

FWIW, I think we should do what I said in https://github.com/mrdoob/three.js/issues/5876#issuecomment-69483293 .

[mrdoob]

I don't see why the API can't allow doing what @jcarpenter wants to do (animate the offset of the alphaMap, while map stays the same). One could do that in canvas, but I think that's a task for the GPU instead.

There is a ton of things one could do by playing with offsets of different maps... offsetting the alphaMap only, scaling the normalMap, etc

Having only one uvTransform per material seems very very limiting.

[mrdoob]

Oh, wait. I think I understand why you guys prefer per material. That way a single vUv gets computed in the vertex shader, instead of computing the uv per pixel in the fragment shader. Right?

[QuaziKb]

per material is ideal because the offsets are actually just implemented using a uniform of the material and have nothing to do with the texture, so we end up with a non-sensical workaround if we need to change the uniform per material that is having to make tons of new objects/textures (which is pretty bad in JS/terrible in WebGL) just to control something that is really just a feature of the material uniforms that are hidden from users. In no way is having it part of texture advantageous, more efficient, or even clearer, and it means the one real application of specifying UVs that change at runtime, animation, is rendered slow and inefficient because of the api.

Textures should specify an image, and anything pertaining to that image. UV offsets have nothing to do with the image / image properties, and everything to do with the material displaying the texture, it's nonsense to even have this feature if its not part of the material, since there needs to be tons of cloning going on per instance, and updating of many textures simultaneously if one wants to actually use the feature in many cases.

imagine one has a different image for every frame of an animated tile, and also wanted to control the offset/repeat of that tile. They'd have to update every incoming frame with the offsets, as well as have copies of every frame for every unique instance of material using that tile, it quickly snowballs into hundreds of new objects and extra assignments, for something that is actually just controlling a few floats per material behind the scenes using all of these objects for no good reason.

as for the transforms per map, although it'd be nice, the uniform cost is too high for no reason in most cases, except if we have a complex API to control when and how the transforms should be unique or shared or dynamic, IMO having that control would be a good thing, but it'd have to be carefully thought out.

[WestLangley]

Oh, wait. I think I understand why you guys prefer per material. That way a single vUv gets computed in the vertex shader, instead of computing the uv per pixel in the fragment shader. Right?

No. The uv is computed in the vertex shader as always.

Imagine a sprite sheet and 20 sprites. Currently, we need 20 cloned materials and 20 cloned textures -- just as in http://threejs.org/examples/misc_ubiquity_test2.html. (BTW, note we already have SpriteMaterial.rotation.)

Moving offset to material, we would need 20 cloned materials and one texture.

In fact, moving offset to sprite, we would need 1 material and 1 texture.

[mrdoob]

Imagine a sprite sheet and 20 sprites. Currently, we need 20 cloned materials and 20 cloned textures

Ohm, I wasn't considering this use case. Thanks!

I'll think about this...

[bhouston]

I advocate not UV transforms in the texture.

For two reasons: (1) it is confusing, (2) there are more events to propagate, and (2) it is wasteful.

Confusion: The reason is we only have a single UV variable for the main maps, but if there is a UV transform on one of the textures, it is confusing that this UV transform may be will be applied to all maps that use the first UV channel, whether or not the other textures have a transform or not. If we did allow for each map to have its own UV transform in the material, I would be more okay with the UV transform associated with the texture -- but that is hard to do because of its fragment uniform usage.

More Event Propagation: The other issue I ran into in Clara.io is the event propagation when trying to animate texture parameters. One needs each texture to keep track of each material that is using it, and then tell those materials that they are dirty and need to be recalculated. It isn't impossible to do this, just more work.

Wasteful: The other issue is that if you have multiple instances of a 3D model or spite and they all have the same animated texture. In that case you would have to have distinct copies of the texture in memory just to have them animated differently -- even though the texture data itself is the same. It is a bit wasteful in that sense as compared to putting the UV transform data on the materials.

Thus if we only have one allowed UV Transform per material, I'd put it on the material itself. I'd follow the Unity 5 model where they have a UV offset, rotation in the material. Game developers are already familiar with this approach.

I think sprites are well handled by UV Transform on the materials as well -- it is very similar to the 3D model case above.

[Zob1]

Having only one uvTransform per material seems very very limiting.

Completely agree here. Not having this functionality is extremely limiting. There are fantastic effects that could be provided here that just aren't because every offset is locked together.

But how can this functionality be made available without choking on clones?

[benaadams]

I think having the values on the Material rather than the texture makes sense for the common use case where all you textures would be matched.

Completely agree here. Not having this functionality is extremely limiting. There are fantastic effects that could be provided here that just aren't because every offset is locked together.

But how can this functionality be made available without choking on clones?

THREE.ShaderMaterial or THREE.RawShaderMaterial would give you this ability, and given it doesn't work currently with regular materials this this the route you'd have to be using anyway.

If you are doing something more funky, it will probably be more funky than just adjusting maps repeats and offsets independently, so you would probably lean this way anyway.

[endel]

+1 for this, guys.

It would be very useful when you have a giant sprite sheet (a.k.a. atlas) and want to reuse it's texture on multiple THREE.Sprite instances.

[QuaziKb]

Any news on this? This problem is still a pretty glaring flaw in the API. Most map types have offsets/repeats that are unused, and event propagation makes something like an animated sprite on a plane way slower/memory heavy than it needs to be.

Flexibility for animated maps DOES NOT EXIST in the current system. The argument keeps coming up that we would be reducing flexibility by tying the settings to the material. This argument is moot because this flexibility doesn't exist in the current system. You can only set the offset/repeat globally for the material, and it's taken from the diffuse map(?). This leads to an even worse problem where there are redundant "offset" / "repeat" settings on most maps in use, and whenever you want to share textures for animation you can't, you need to make a clone, so flexibility is significantly reduced. You expect each texture / map to have unique offsets but this isn't possible as things stand, and in most cases you actually want one set of UV offsets because it'd be annoying to set the offsets to the same thing for normal/spec/diffuse (a scrolling normal map over a fixed diffuse map is a niche use where a shader material could be used).

If you look at the actual shaders being built, the texture offset / repeat IS tied to the material, but strangely copied off of one map which in no way should be in control. The material NEEDS to be in control for this to be fast and elegant without redundancy.

The best of both world is possible with tons of extra flags, but i don't think this is a better solution than just directing users to try using the shader material instead for these kinds of specific edge cases.

[bhouston]

The solution to this is @sunag's NodeMaterial btw.

[karimbeyrouti]

Also adding my +1 for this ! Would make working with sprite sheets much nicer.

[rhys-vdw]

:+1: I found this surprising. I've had to start duplicating my textures for each sprite in game, as all my repeated sprites were animating each other. Sounds like this means I'm loading redundant sprite data to the GPU?

Does anybody have a workaround for this issue? Might it be possible to manipulate the UV offset by directly setting the uniform on the shader, bypassing the Texture interface?

The solution to this is @sunag's NodeMaterial btw.

@bhouston, could you please provide a link? There is no public repository under @sunag's account with that name.

[bhouston]

@rhys-vdw It is located here in the dev branch only:

https://github.com/mrdoob/three.js/tree/dev/examples/js/nodes

It is new so I am not sure if it is ready to use on sprites, but it a fully graph-based shader system, so it will give you the flexibility to modify texture accesses arbitrarily.

[sunag]

It is new so I am not sure if it is ready to use on sprites, but it a fully graph-based shader system, so it will give you the flexibility to modify texture accesses arbitrarily.

@bhouston I can create an example with Node, this looks good.

About the THREE.SpriteMaterial you could access offset/scale for create the spritesheet with this for example:

var offsetX = frameX / mapWidth;
var scaleX = mapWidth / frameWidth;

sprite.material.map.offset.x = offsetX;
sprite.material.map.repeat.x = scaleX;

https://github.com/mrdoob/three.js/blob/master/src/renderers/webgl/plugins/SpritePlugin.js#L53

[QuaziKb]

The node based system isnt a fix... It still doesnt address the issues with the api. It takes 2 seconds to add offset/repeat to the material prototype, and make the renderer read off of that instead of the texture. Ive done it already for my project, but the fact remains its an obvious flaw in the api that should be officially changed to prevent headaches for new users who WILL run into this issue if they're trying to do something common.

[rhys-vdw]

...so it will give you the flexibility to modify texture accesses arbitrarily.

tbh I don't know what that means. To animate individual set shader uniforms for "offset" and "repeat" on a per material basis.

About the THREE.SpriteMaterial you could access offset/scale for create the spritesheet with this for example...

@sunag, perhaps the issue is not clear. The code you've shared mutates the texture, which is shared by all instances of that material. This means that it is impossible to have two materials sharing a texture, but with unique offsets - for example, two enemy sprites showing different animation frames.

Ive done it already for my project, but the fact remains its an obvious flaw in the api that should be officially changed to prevent headaches for new users who WILL run into this issue if they're trying to do something common.

@QuaziKb Is there a PR that I can target for my project?

Although this whole thing wouldn't be a problem if, as @WestLangley said, the following were true:

If we keep the current approach, one thing that needs to be changed, is when a texture is cloned, the shared imaged should only be passed to the GPU once.

Is that correct?

[sunag]

@sunag, perhaps the issue is not clear. The code you've shared mutates the texture, which is shared by all instances of that material. This means that it is impossible to have two materials sharing a texture, but with unique offsets - for example, two enemy sprites showing different animation frames.

Hmm, for this NodeMaterial certainly would solve, you will be able to share the same texture with different materials and independent uv offset and advantages like customized filters and other things that a node-based material can offer.

https://github.com/mrdoob/three.js/issues/7522

But at the moment someone tried to instantiate the uuid like this:?

THREE.Texture.prototype.createInstance = function() {

    var inst = this.clone();

    inst.uuid = this.uuid;
    inst.version = this.version;

    return inst;

}

if you use needsUpdate update all instances version too.