Do I misunderstand how pipelined, immutable textures work?

[markebowles]

This is not so much an issue report as a request for assistance. I've been working with GPU.js for a month or so, and think I understand it, but I'd appreciate help with a seemingly mysterious behavior.

I'm building a classic multi-layer image compositor using GPU.js. The code assembles a stack of images as layers, assigns each a Z (depth) value, and then sequentially overlays them into a composite image, that is finally displayed in a browser canvas.

I'm confident through testing that the layer images are correct. The central composition loop looks like this:

    compose(allLayers, viewport) {
        // filter out inelligible layers and sort them by decreasing depth
        const layers = allLayers.filter((layer) => (!layer.muted) && 'annot' in layer && 'bounds' in layer.annot);
        layers.sort((layer1, layer2) => { return (layer1.z - layer2.z) });

        // GPU kernel pipelines an image of [0,0,0,0] pixels
        let img1 = this.clear();

        layers.forEach((layer) => {
            // GPU kernel overlays this layer's non-transparent pixels into the image
            const img2 = this.overlayLayer(img1, layer, viewport);
            img1.delete();
            img1 = img2;
        })

        // graphical GPU kernel writes the result to the canvas using this.color()
        this.toViewport(img1, [0.7, 0.7, 0.7, 1.0]);
        img1.delete();
    }

The three GPU.js kernels in this part of the process are: clear(), that returns a texture of transparent pixels; overlayLayer(), that for each pixel in the image, makes the compositing decision and returns another texture with the composition, and toViewport(), that draws the input texture to the canvas.

The compose() method is called from a requestAnimationFrame call in a Chrome browser in mac os. clear() and overlayLayer() are pipelined and immutable. All three kernels are associated with the same canvas, and their output specification is the 2d width/height of the canvas.

With a few simple layers, this works properly. Some of the layers are actually video html elements, and the compositing is fast enough to keep up. But, with more layers, mysterious things happen. Some of the layers don't appear at all, and in some cases, the clear() kernel seems to stop working, and the resulting canvas gets painted with multiple frames. The misbehavior does not seem to be performance or load-dependent. When it happens, it's consistent and deterministic.

My question concerns the use of pipelined, immutable textures. I envision the textures, as they appear in a javascript program, as remote pointers to blocks of data in the GPU. So, the statement "img1 = img2" can be interpreted as "copy the pointer to the output texture to the javascript variable 'img1', removing the reference to the previous texture, that we have just deleted."

My suspicion is that I've got this wrong somehow, and there's "cross-talk" between the textures or the kernels that I'm not aware of. Or, that there's some other reason why the sequencing in this little method is somehow invalid.

Assistance will be hugely appreciated!

[markebowles]

This is a screenshot of the output canvas attached to the compositor. The guy in the red checked shirt comes from a green screen test video. He raises his arm and does a "thumbs up" over a 15-second period. The red, blue and yellow panels are static layers used to test the z-ordering. The fact that we are seeing accumulation of image data over all the frames -- in spite of the "clear()" kernel that's supposed to start the composition with an empty image on each frame -- makes me suspect there's a basic assumption in this code that's wrong.