bindFramebuffer reset with "null" behaviour

[sejori]

Hi there, great work on your library - thanks :)

We have an issue when creating new textures and Framebuffers to store off-canvas data. The issue is that we cannot get the render target to return to the default texture after calling gl.bindFramebuffer and gl.framebufferTexture2D.

Here is our FBO class:

import 'package:flutter_angle/flutter_angle.dart';

class FBO {
  final RenderingContext gl;
  final int texId;
  final int w;
  final int h;
  final int internalFormat;
  final int format;
  final int type;
  final int param;

  late WebGLTexture texture;
  late Framebuffer fbo;

  FBO({
    required this.gl,
    required this.texId,
    required this.w,
    required this.h,
    required this.internalFormat,
    required this.format,
    required this.type,
    required this.param,
  }) {
    gl.activeTexture(WebGL.TEXTURE0 + texId);

    texture = gl.createTexture();

    gl.bindTexture(WebGL.TEXTURE_2D, texture);
    gl.texParameteri(WebGL.TEXTURE_2D, WebGL.TEXTURE_MIN_FILTER, param);
    gl.texParameteri(WebGL.TEXTURE_2D, WebGL.TEXTURE_MAG_FILTER, param);
    gl.texParameteri(
        WebGL.TEXTURE_2D, WebGL.TEXTURE_WRAP_S, WebGL.CLAMP_TO_EDGE);
    gl.texParameteri(
        WebGL.TEXTURE_2D, WebGL.TEXTURE_WRAP_T, WebGL.CLAMP_TO_EDGE);
    gl.texImage2D(
        WebGL.TEXTURE_2D, 0, internalFormat, w, h, 0, format, type, null);

    fbo = gl.createFramebuffer();

    gl.bindFramebuffer(WebGL.FRAMEBUFFER, fbo);
    gl.framebufferTexture2D(WebGL.FRAMEBUFFER, WebGL.COLOR_ATTACHMENT0,
        WebGL.TEXTURE_2D, texture, 0);
    gl.viewport(0, 0, w, h);
    gl.clear(WebGL.COLOR_BUFFER_BIT);
  }
}

And here is the initPlatformState from our test implementation, based on your example:

  void blit(RenderingContext gl, {Framebuffer? destination}) {
    gl.bindFramebuffer(WebGL.FRAMEBUFFER, destination);
    gl.drawElements(WebGL.TRIANGLES, 6, WebGL.UNSIGNED_SHORT, 0);
  }

  // Platform messages are asynchronous, so we initialize in an async method.
  Future<void> initPlatformState() async {
    didInit = true;
    final mq = MediaQuery.of(context);
    screenSize = mq.size;
    dpr = mq.devicePixelRatio;

    await FlutterAngle.initOpenGL(true);

    final options = AngleOptions(
      width: 1000,
      height: 1000,
      dpr: dpr,
    );

    try {
      textures.add(await FlutterAngle.createTexture(options));
    } on PlatformException catch (e) {
      print("failed to get texture id $e");
      return;
    }

    final _gl = textures[0].getContext();

    final velocityFBO = FBO(
      gl: _gl,
      texId: 10,
      w: 1000,
      h: 1000,
      internalFormat: WebGL.RGBA,
      format: WebGL.RGBA,
      type: WebGL.HALF_FLOAT,
      param: WebGL.LINEAR,
    );

    blit(_gl, destination: null); // <- attempt to reset target texture / buffer

    /// This is the base vertex buffer that will be used for all the programs
    _gl.bindBuffer(WebGL.ARRAY_BUFFER, _gl.createBuffer());
    _gl.bufferData(WebGL.ARRAY_BUFFER,
        Float32Array.fromList([-1, -1, -1, 1, 1, 1, 1, -1]), WebGL.STATIC_DRAW);
    _gl.bindBuffer(WebGL.ELEMENT_ARRAY_BUFFER, _gl.createBuffer());
    _gl.bufferData(WebGL.ELEMENT_ARRAY_BUFFER,
        Uint16Array.fromList([0, 1, 2, 0, 2, 3]), WebGL.STATIC_DRAW);
    _gl.vertexAttribPointer(0, 2, WebGL.FLOAT, false, 0, 0);
    _gl.enableVertexAttribArray(0);

    splatProgram = GlProgram(
      'Splat Program',
      _gl,
      Fragment.splat,
      Fragment.baseVertex,
    );

    splatProgram?.logValues();
    splatProgram?.bind();

    _gl.uniform3f(splatProgram!.uniforms['color']!, 0.25, 0.5, 0.3);
    _gl.uniform2f(splatProgram!.uniforms['point']!, 0.3, 0.2);
    _gl.uniform1f(splatProgram!.uniforms['radius']!, 0.002);
    _gl.uniform1i(splatProgram!.uniforms['uTarget']!, textures[0].textureId);
    blit(_gl, destination: null);

    if (!mounted) return;
    setState(() {
      textureId = textures[0].textureId;
    });

    ticker = createTicker(updateTexture);
    ticker.start();
  }

  // rest of file follows example...

With the velocityFBO instantiation the screen goes white but no errors show. When we remove the call to FBO we successfully render the "splat" fragment. We can also see that if we comment out the following two lines from the FBO constructor we can continue to render as expected:

    gl.bindFramebuffer(WebGL.FRAMEBUFFER, fbo);
    gl.framebufferTexture2D(WebGL.FRAMEBUFFER, WebGL.COLOR_ATTACHMENT0,
        WebGL.TEXTURE_2D, texture, 0);

Which leads me to believe the issue lies in binding new frame buffers and textures to the canvas.

[Knightro63]

Hi @sejori,

Thanks for reaching out and using this repo. I have taken a look at the above code and without some of the other portions I wont be able to help completely.

The bindFramebuffer and framebufferTexture2D are not need a lot of the time due to have a different fbo that is attached that angle itself calls to render most items. When you call them they are making new ones that will need to have a different source texture to get attached to then rendered after calling the activate texture then updateTexture.

This and this are other more complicated ways to render using this package.

If you give me more of the code I can see if I can help. I would need the updateTexture function and splatProgram.

Sorry I am unable to help, but please see if you can give me more info.

Thanks

[alexgrusu]

Hey @Knightro63

Thanks for your response.

I uploaded our code here

It would be great if we could hear your thoughts and confirm that the usage of the library is correct.

[Knightro63]

Hi @sejori and @alexgrusu,

I submitted a pull request to the repo provided and I think I found your issue.

Let me know if you have any other questions.

[alexgrusu]

Hey @Knightro63 ,

Thanks for your valuable feedback. This was helpful.

We are running into an app freeze after running the shaders animation for a while.

I see you are calling _rc.flush(); on every update, is there anything else we need to clear from time to time?

Also, we see that the animation is a bit blurry. Is this something we can fix in any way?

Screenshot for the blurry effect

[Knightro63]

Hi @alexgrusu,

I looked at you demo again and noticed that some of your buffers were called continuously and not deleted everytime which would cause a memory leak. This is probably why your app would freeze after some time. I was able to find some of them, but you would need to go in and find the rest.

As for the blurry animation, this can be fixed by binding your texture to a larger size than your width and height. Then rebinding it to the correct size, kind of a janky way to increase your resolution. Here is a form for which they answered this.

Hope this helps.

[alexgrusu]

Hi @alexgrusu,

I looked at you demo again and noticed that some of your buffers were called continuously and not deleted everytime which would cause a memory leak. This is probably why your app would freeze after some time. I was able to find some of them, but you would need to go in and find the rest.

As for the blurry animation, this can be fixed by binding your texture to a larger size than your width and height. Then rebinding it to the correct size, kind of a janky way to increase your resolution. Here is a form for which they answered this.

Hope this helps.

Hey @Knightro63 ,

I'm extremely grateful for your help.

Just to confirm my understanding, maybe my question is dumb because I don't know what is happening behind the scenes.

Whenever we create a buffer, bind, and draw the elements, at the end we need to call deleteBuffer. Is my understanding correct?

Are there any other scenarios when we need to call deleteBuffer? Or what's the sequence of calls we need to follow to ensure we don't lead to any memory leaks?

[Knightro63]

Hi @alexgrusu,

Yes this is true. In your bilt function you create 2 buffers each time but they are never deleted afterwards.

If you go to my fork I have updated it with the correct bilt function.

For anything that is created for the gles program you will need to delete them after they are done. This is true for framebuffers, textures, buffers, vertexarrays, ect. In most of your case it looks correct all of the items you create you use over and over, which will not need to be deleted for each update, but they will need to be deleted if you go to a different portion of you app.

There is still one in there that I was unable to find; this is causing a small memory leak.

Hope this helps.

[alexgrusu]

Hey @Knightro63 ,

How do you investigate the memory leaks? Do you use flutter dev tools or something else?

Is it possible to use something like Valgrind?

Also, do you have any thoughts about the following error?

E/emuglGLESv2_enc( 8847): device/generic/goldfish-opengl/system/GLESv2_enc/GL2Encoder.cpp:s_glDrawElements:1479 GL error 0x506 condition [ctx->m_state->checkFramebufferCompleteness(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE]

[Knightro63]

Hi @alexgrusu,

For memory leaks I use XCode. I know you are using an Android device and the flutter dev tools should be able to show you the memory information as well.

For the error it might be due to checkFramebufferCompleteness not being implemented at this time.

[alexgrusu]

Hey @Knightro63 ,

Considering we run through different contexts across the app, is the FlutterAngle.initOpenGL call required every time before running the GL work, or do we have to call this only once during the lifetime of the app?

Also, I don't see any support to release/dispose resources to OpenGL. Is this required?

[Knightro63]

Hi @alexgrusu,

Yes you only need to call initOpenGL once. You can call it more but it will only use the initially created library.

If you want to remove the texture from FlutterAngle call FlutterAngle.deleteTexture.

Hope this helps.