Texture SubImage Interface

[apetrock]

The current subimage interface is a little clunky:

renderer.copyTextureToTexture ( position : Vector2, srcTexture : Texture, dstTexture : Texture, level : Number ) : null

This means that in order to move textures around between atlases you have to pass a reference to a renderer around.

I worked with one of the original developers of OpenSceneGraph and I really liked the solution he used. The suggestion is to move the subimage interface to the texture. Each texture would have an array of SubImageData that can be appended every frame by the user when data becomes available. Subimages get applied during rendering, then the SubImageData array gets flushed.

I worked up the idea in a dev build, but I'm a bit remiss to check it in because my implementation was far from perfect, and its such a small inclusion, a feature request seemed like a better way to discuss the idea.

here is the code I put together:

in renderers/WebGlTextures.js:

      function subImage2D( textureProperties, texture, slot ) {
        state.activeTexture( 33984 + slot );
        state.bindTexture( 3553, textureProperties.__webglTexture );

        _gl.pixelStorei( 37440, texture.flipY );
        _gl.pixelStorei( 37441, texture.premultiplyAlpha );
        _gl.pixelStorei( 3317, texture.unpackAlignment );

        if ( texture.subImageData.length > 0 ) {

            _gl.bindTexture( 3553, textureProperties.__webglTexture );

            for ( var i = 0; i < texture.subImageData.length; i ++ ) {

                var data = texture.subImageData[ i ];
                var glFormat = utils.convert( data.format );
                var glType = utils.convert( data.type );

                _gl.texSubImage2D( 3553,
                    data.level,
                    data.offset[ 0 ],
                    data.offset[ 1 ],
                    glFormat,
                    glType,
                    data.pixels );

            }

        }

        texture.subImageData = [];
        texture.needsUpdate = false;

    }

I added another elseif in the setTexture2D:

    function setTexture2D( texture, slot ) {

        var textureProperties = properties.get( texture );

        if ( texture.isVideoTexture ) updateVideoTexture( texture );

        if ( texture.version > 0 && textureProperties.__version !== texture.version ) {

            var image = texture.image;

            if ( image === undefined ) {

                console.warn( 'THREE.WebGLRenderer: Texture marked for update but image is undefined' );

            } else if ( image.complete === false ) {

                    console.warn( 'THREE.WebGLRenderer: Texture marked for update but image is incomplete' );

            } _else if(texture.subImageData.length > 0){
                                        subImage2D(textureProperties, texture, slot);
                                        return;
            } else if(!texture.ignoreImage){

                uploadTexture( textureProperties, texture, slot );
                return;

            }_

        }

        state.activeTexture( 33984 + slot );
        state.bindTexture( 3553, textureProperties.__webglTexture );

    }

in the texture class:

           /* in the "constructor" */
    this.subImageData = [];
    this.ignoreImage = false;

           /*added method*/
    subImage : function(level, offsetX, offsetY,  pixels, format, type){

        format = format !== undefined ?  format : this.format;
        type   = type   !== undefined ?  type   : this.type;

        if(type !== this.type){
            console.warn('THREE.Texture.subimage: type should equal texture type, bailing');
            return;
        }

        if(format !== this.format){
            console.warn('THREE.Texture.subimage: format should equal texture format, bailing');
            return;
        }

        /*
        once you subdata, you have to ignore the image that is attached to this texture
        seems a little clunky, but I can't think of another way to do it, unless there was
        a specific subImageTexture that didn't bind an image
         */        

        this.ignoreImage = true;
        this.subImageData.push({
            level: level,
            offset: [offsetX,offsetY],
            format: format,
            type: type,
            pixels: pixels
        });
    },

Right now I'm using this with an empty DataTexture, and append images as they come in from the server. An image gets pushed into the atlas and I'm using it like this:

        desTexture.subImage(0, offsetIntoDestX, offsetIntoDestY, sourceTexture);

[DavidPeicho]

I could work on this if we agree on what kind of API we want to expose to the user. I also make good use of the texSubImage* feature, but I don't have access to the renderer, only to the texture itself.

We could go for something simple like an array of sub images like you proposed, but instead of applying it on every frame, why not apply it only during a texture upload? In a nutshell, if some subimages data are given, we use the texSubImage* call with those data instead in WebGLTextures.upload().

If someone needs to do that every frame no matter what happened, I guess it's better to directly for them to hook themselves in onBeforeRender() or a similar method.

[apetrock]

Hi David,

I have a mirror that I've worked on where I've broadened the implementation a little bit.

I'm not sure that you can view this, our public repo is not up to date with this, right now: https://gitlab.com/viqi/mirrors/three.js/-/blob/develop/src/renderers/webgl/WebGLTextures.js

The gist is that the renderer looks at the array of subimages on the data texture, and if there is a new subimage available, it applies the subimage then deletes the reference. I worked with one of the original writers of Open Scene Graph and this is how we did it. It won't get applied every frame, unless there is a new subimage available.

On Fri, Dec 4, 2020 at 7:35 AM David Peicho notifications@github.com wrote:

I could work on this if we agree on what kind of API we want to expose to the user. I also make good use of the texSubImage* feature, but I don't have access to the renderer, only to the texture itself.

We could go for something simple like an array of sub images like you proposed, but instead of applying it on every frame, why not apply it only during a texture upload? In a nutshell, if some subimages data are given, we use the texSubImage* call with those data instead in WebGLTextures.upload().

If someone needs to do that every frame no matter what happened, I guess it's better to directly for them to hook themselves in onBeforeRender() or a similar method.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/17162#issuecomment-738850386, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLDQLJ2JDYYDZTFTUMIDSTSTD6VDANCNFSM4IJBJJYA .

[apetrock]

Here is the snippet applied on a texture array:

function subImage2DArray(textureProperties, texture, slot) {

state.activeTexture(33984 + slot); state.bindTexture(35866, textureProperties.webglTexture); _gl.pixelStorei(37440, texture.flipY); _gl.pixelStorei(37441, texture.premultiplyAlpha); _gl.pixelStorei(3317, texture.unpackAlignment); if (texture.subImageData.length > 0) { _gl.bindTexture(35866, textureProperties.webglTexture); for (var i = 0; i < texture.subImageData.length; i++) { var data = texture.subImageData[i]; var glFormat = utils.convert(data.format); var glType = utils.convert(data.type); _gl.texSubImage3D(35866, data.level, data.offset[0], data.offset[1], data.offset[2], data.pixels.width, data.pixels.height, 1, glFormat, glType, data.pixels); } } texture.subImageData = []; texture.needsUpdate = false; }

else if (texture.isDataTexture) { // use manually created mipmaps if available // if there are no manual mipmaps // set 0 level mipmap and then use GL to generate other mipmap levels if (mipmaps.length > 0 && supportsMips) { for (let i = 0, il = mipmaps.length; i < il; i++) { mipmap = mipmaps[i]; state.texImage2D(_gl.TEXTURE_2D, i, glInternalFormat, mipmap.width, mipmap.height, 0, glFormat, glType, mipmap.data); } texture.generateMipmaps = false; textureProperties.maxMipLevel = mipmaps.length - 1; } else if (texture.subImageData.length > 0) { subImage2D(textureProperties, texture, slot); } else { state.texImage2D(_gl.TEXTURE_2D, 0, glInternalFormat, image.width, image.height, 0, glFormat, glType, image.data); textureProperties.maxMipLevel = 0; } }

On Fri, Dec 4, 2020 at 9:47 AM John Delaney a.pet.rock@gmail.com wrote:

Hi David,

I have a mirror that I've worked on where I've broadened the implementation a little bit.

I'm not sure that you can view this, our public repo is not up to date with this, right now:

https://gitlab.com/viqi/mirrors/three.js/-/blob/develop/src/renderers/webgl/WebGLTextures.js

The gist is that the renderer looks at the array of subimages on the data texture, and if there is a new subimage available, it applies the subimage then deletes the reference. I worked with one of the original writers of Open Scene Graph and this is how we did it. It won't get applied every frame, unless there is a new subimage available.

On Fri, Dec 4, 2020 at 7:35 AM David Peicho notifications@github.com wrote:

I could work on this if we agree on what kind of API we want to expose to the user. I also make good use of the texSubImage* feature, but I don't have access to the renderer, only to the texture itself.

We could go for something simple like an array of sub images like you proposed, but instead of applying it on every frame, why not apply it only during a texture upload? In a nutshell, if some subimages data are given, we use the texSubImage* call with those data instead in WebGLTextures.upload().

If someone needs to do that every frame no matter what happened, I guess it's better to directly for them to hook themselves in onBeforeRender() or a similar method.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/17162#issuecomment-738850386, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLDQLJ2JDYYDZTFTUMIDSTSTD6VDANCNFSM4IJBJJYA .

[DavidPeicho]

I can't access the code but I understand what you did. Personally I wouldn't modify subImageData at all. First because something like that:

texture.subImageData = []

Makes a new allocation. However, I like the idea of having just an array of sub image to apply. I think it's just a valid point to say, if there are things in the array, upload uses texSubImage instead of re-uploading everything.

My second idea is to go for something much simpler, like an update method where the user can simply override the behavior of upload... It's more for advanced user though.

Do you think that's a valid change I could bring to WebGLTextures? @Mugen87 @mrdoob Let met know what you think is the best API-wise. I think it's just a little hard to have the subimage udpate only on the renderer, because the texture are always renderer-agnostic.

[apetrock]

In my implementation, I'm using DataTextures as an image pool. So some sort of flushing of texture.subImageData after it has been applied is necessary, whether that is user controlled or handled automatically by the renderer is up for a debate, but I don't see any reason to keep the contents of the array alive after subimages have been applied since they will be stored in texture on the GPU. "texture.subImageData = []" is expedient and javascript garbage collection handles the rest. I suppose popping the elements of the SubimageData array instead of setting it to an empty array is a valid alternative. Otherwise it is up to the user to make sure that stale data doesn't remain in the array before new data is added. Per your last comment, prior to using subimage, I was doing the same thing with the Canvas API: drawImage. So a non OpenGl implementation is very easy to add. This is what subimage looks like in Texture.js: subImage: function(level, offset, pixels, format, type) {

    format = format !== undefined ? format : this.format;
    type = type !== undefined ? type : this.type;

    if (type !== this.type) {
        console.warn(

'THREE.Texture.subimage: type should equal texture type, bailing'); return; }

    if (format !== this.format) {
        console.warn(

'THREE.Texture.subimage: format should equal texture format, bailing'); return; }

    this.ignoreImage = true;
    this.subImageData.push({
        level: level,
        offset: offset,
        format: format,
        type: type,
        pixels: pixels
    });
},

On Sun, Dec 6, 2020, 8:40 AM David Peicho notifications@github.com wrote:

I can't access the code but I understand what you did. Personally I wouldn't modify subImageData at all. First because something like that:

texture.subImageData = []

Makes a new allocation. However, I like the idea of having just an array of sub image to apply. I think it's just a valid point to say, if there are things in the array, upload uses texSubImage instead of re-uploading everything.

Do you think that's a valid change I could bring to WebGLTextures? @Mugen87 https://github.com/Mugen87 @mrdoob https://github.com/mrdoob Let met know what you think is the best API-wise. I think it's just a little hard to have the subimage udpate only on the renderer, because the texture are always renderer-agnostic.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/17162#issuecomment-739528332, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLDQLLT3WASRZ4JPLCCTGDSTOXY5ANCNFSM4IJBJJYA .

[DavidPeicho]

You can't really flush anything from a Texture object because the object itself isn't renderer-dependant. So let's say someone re-use the texture between two renderes, imagine two canvases displaying the same thing but with different cameras, you will only update the webgl object on one of the renderer and not the other.

[apetrock]

Ah, I see what you mean. But once you apply the subdata to the opengl texture object, then that data will be present on the GPU, so any other renderer should also be displaying the updated data, does Threejs not support sharing data between contexts? If for whatever reason the other renderer isn't opengl, then that does present a problem.

On Mon, Dec 7, 2020 at 11:29 AM David Peicho notifications@github.com wrote:

You can't really flush anything from a Texture object because the object itself isn't renderer-dependant. So let's say someone re-use the texture between two renderes, imagine two canvases displaying the same thing but with different cameras, you will only update the webgl object on one of the renderer and not the other.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/17162#issuecomment-740130204, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLDQLNIEHKMLCPTJYEC3VTSTUUIDANCNFSM4IJBJJYA .

[DavidPeicho]

It doesn't work like that unfortunately. WebGL contexts are isolated. Three.js keeps cached GPU objects per renderer instance. Even if they wanted to support sharing data, they actually wouldn't be able to. It may be solved in the WebGPU specification but unfortunately isn't possible anyway with WebGL.

[DavidPeicho]

I think it would be great to discuss a bit this issue because that's something that can be improved for WebGPU for instance.

Resources are uploaded lazily per context, but in some cases, we need to apply per-context transformation on a resource. This is going to be even more true with WebGPU that will allow more resource modification.

I was thinking, wouldn't it be better to have access to the list of contexts a resource is uploaded in? Right now, context and resource are bound by the dispose() method, but I think it could be useful to have something like:

const texture = new DataTexture();
texture.uploadSubImage(...);

And DataTexture would look like:

class DataTexture {

    uploadSubImage(...) {

        for (const context of this._contexts) {
            context.subImage2D(...);
        }

    }

}

Obviously, context should expose an API that's compatible between WebGL and WebGPU... I am not a fan of having references to contexts like that, but that's the best way I could come up with to have a way for a user to easily modify a resource.

In my use case, sometimes I deal with several contexts and some of them share the same resource object. However, just from the object I can't really retrieve the context to perform those kind of operations.

@mrdoob

[apetrock]

Maybe instead of holding onto a context, one could use resource counting/hash on the subimage struct. You can free the subimage structure once all of the contexts have been hit. Maybe guarding against multiple applications of the subimage to the same context using a guard on struct based on the context id.

a few alternatives: 1) might be to bind the context in a callback somewhere in the rendering pipeline. So the texture wouldn't have a reference to the context, but it would be available through binding.

2) a pre-rendering stage that can detect if any global changes have occured and handle any global changes to the context.

I'm just spitballing here.

On Wed, Jan 27, 2021 at 9:52 AM David Peicho notifications@github.com wrote:

I think it would be great to discuss a bit this issue because that's something that can be improved for WebGPU for instance.

Resources are uploaded lazily per context, but in some cases, we need to apply per-context transformation on a resource. This is going to be even more true with WebGPU that will allow more resource modification.

I was thinking, wouldn't it be better to have access to the list of contexts a resource is uploaded in? Right now, context and resource are bound by the dispose() method, but I think it could be useful to have something like:

const texture = new DataTexture();texture.uploadSubImage(...);

And DataTexture would look like:

class DataTexture {

uploadSubImage(...) {

    for (const context of this._contexts) {
        context.subImage2D(...);
    }

}

}

Obviously, context should expose an API that's compatible between WebGL and WebGPU... I am not a fan of having references to contexts like that, but that's the best way I could come up with to have a way for a user to easily modify a resource.

In my use case, sometimes I deal with several contexts and some of them share the same resource object. However, just from the object I can't really retrieve the context to perform those kind of operations.

@mrdoob https://github.com/mrdoob

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/17162#issuecomment-768461756, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLDQLLVECTWBW7W35APQD3S4BHHLANCNFSM4IJBJJYA .

[DavidPeicho]

Freeing when all context have been triggered is fine, but then you still don't know what context has this resource... For the case of changing a SSBO, chaing subregions of textures, etc...

[DavidPeicho]

I think the easiest would still be a GraphicsContext that is bound to either a WebGL context or a WebGPU context, and that does the interface between them. A resource could be bound to several GraphicsContext and executing a manipulation of the resource basically manipulates them all