IOS: Multiple UI Views

[JosephAustin]

I know bgfx can render to multiple windows on desktop, but I haven't been able to find a way to do it for multiple UIView instances.

It works fine to set a single window's CAEGLLayer as an nwh in the platform data, but that locks it onto a specific view, and there's no way to swap context or nwh later. As shown in the next post, I am trying to find a way to use custom framebuffers for this need as per the windowing demo.

[JosephAustin]

I have tried a lot of different things by now. I'm almost certain I have bgfx rendering offscreen to a bunch of different framebuffers now, but trying to get those to render to different CAEAGLLayers isn't working.

Following the windowing example:

for(MyGameView * view in _viewInstances)
    {
        CGFloat w = [view bounds].size.width * [[UIScreen mainScreen] scale];
        CGFloat h = [view bounds].size.height * [[UIScreen mainScreen] scale];

        if((w > 0) && (h > 0))
        {
            // Make sure we have a valid frame buffer loaded for this view
            bgfx::FrameBufferHandle fbh = [view frameBuffer];
            if(fbh.idx == bgfx::kInvalidHandle )
            {
                fbh = bgfx::createFrameBuffer((__bridge void*)[view layer], uint16_t(w), uint16_t(h));
                [view setFrameBuffer:fbh];
            }

            bgfx::setViewFrameBuffer(iteration, fbh);
            bgfx::setViewRect(iteration, 0, 0, uint16_t(w), uint16_t(h) );
            bgfx::setViewClear(iteration
                               , BGFX_CLEAR_COLOR|BGFX_CLEAR_DEPTH
                               , 0xffff00ff
                               , 1.0f
                               , 0
                               );
            bgfx::touch(iteration);
            iteration += 1;
        }
    }

    if(iteration > 1) {
        bgfx::frame();

        for(MyGameView * view in _viewInstances) {
            [view render];
        }
    }

So now I'm almost sure I have all my renders sitting in frame views internally, but I can't seem to use [context renderbufferStorage:GL_RENDERBUFFER fromDrawable:_eaglLayer]; to swap render targets. Of course to create the frame buffers i passed in the different eagl layers, so I was surprised that [context presentRenderbuffer:GL_RENDERBUFFER]; wasn't enough.

[JosephAustin]

Addtl note: I can't appear to make bgfx use multiple rendering contexts so I'm doing this

        bgfx::PlatformData platformData;
        platformData.context = (__bridge void*)_context;
        bgfx::setPlatformData(platformData);
        bgfx::init(bgfx::RendererType::OpenGLES);

and trying to share all the rendering windows through that one context, while preserving single-threadedness

Not doing platform data leads to a crash in BGFX_MUTEX_SCOPE(m_resourceApiLock); (bad access)

[bkaradzic]

bgfx::PlatformData should be used only for primary/default window. For other windows you should use bgfx::createFrameBuffer from native window handle. If your code works on desktop but fails on iOS there is some iOS specific issue with it (I haven't tested that code path, but I assume that whoever submitted PR for that change did).

[JosephAustin]

I see. That's tricky since there is no primary window; you would ideally be able to create and destroy as many UIViews of this type as the app demands. I'll see if I can make something like that work, though.

[bkaradzic]

Primary window is required on some platforms, at some point I might change that it's required to call bgfx::createFrameBuffer for even primary window if bgfx::PlatformData is not set, but it's not how it's working right now.

[JosephAustin]

Well that wouldn't fix it either sadly, the issue is that i went into this trying to make a View for use in ios and android. I would need some way to be able to target multiple contexts or EAGLLayers, with no promises that they wont get added or deleted in any order. More like widgets than windows, really.

I'm attempting to have some kind of off-screen hidden window and seeing if that works. If not, I may try hacking at bgfx and worst case, I'll give up and do opengl.

[bkaradzic]

Why you don't create render target and render it there?

[JosephAustin]

You mean a backbuffer/texture instead of framebuffer?

[JosephAustin]

Good idea... let me try that :) If i get this working I'll definitely post something people can use if anyone else has this issue

[bkaradzic]

No I meant regular frame buffer texture, not back buffer. :)

https://bkaradzic.github.io/bgfx/bgfx.html#_CPPv2N4bgfx17createFrameBufferE8uint16_t8uint16_tN13TextureFormat4EnumE8uint32_t

[JosephAustin]

Oh I see. So I would load up several 'views' to render to custom framebuffers and then bind and display them in each EAGLLayer... will try that

[JosephAustin]

It's not working but I'm sure I'm doing it wrong. I know I'm a bother. Unless I do not understand correctly though, the normal structure for doing multi-window isn't working on ios.

I tried making a 'boss' and 'lackey' instance to test:

- (id)initWithFrame:(CGRect)frame
{
    self = [super initWithFrame:frame];
    if(self) {
        _context = [[EAGLContext alloc] initWithAPI: kEAGLRenderingAPIOpenGLES3];
        [EAGLContext setCurrentContext:_context];

      // I HAVE to do this before initializing bgfx or it doesn't work at all
        glGenRenderbuffers(1, &_colorRenderBuffer);
        glBindRenderbuffer(GL_RENDERBUFFER, _colorRenderBuffer);
        glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
                                  GL_RENDERBUFFER, _colorRenderBuffer);
        [_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(CAEAGLLayer *)self.layer];

        if(boss == nil) {
            boss = self;
            bgfx::PlatformData platformData;
            platformData.nwh = (__bridge void*)render_layer;
            bgfx::setPlatformData(platformData);
            bgfx::init(bgfx::RendererType::OpenGLES);
        } else {
            lackey = self;
        }
    }
    return self;
}

Then I render them together like:

        bgfx::reset(uint16_t(w), uint16_t(h));
        bgfx::setViewRect(0, 0, 0, uint16_t(w), uint16_t(h));
        bgfx::setViewClear(0
                           , BGFX_CLEAR_COLOR|BGFX_CLEAR_DEPTH
                           , 0xffff00ff
                           , 1.0f
                           , 0
                           );

        bgfx::touch(0);

        static bgfx::FrameBufferHandle fbh = bgfx::createFrameBuffer((__bridge void*)lackey.layer, w, h);
        bgfx::resetView(1);
        bgfx::setViewFrameBuffer(1, fbh);
        bgfx::setViewRect(1, 0, 0, uint16_t(w), uint16_t(h));
        bgfx::setViewClear(1
                           , BGFX_CLEAR_COLOR|BGFX_CLEAR_DEPTH
                           , 0xff0000ff
                           , 1.0f
                           , 0
                           );

        bgfx::touch(1);

        bgfx::frame();

        [[boss context] presentRenderbuffer:GL_RENDERBUFFER];
        [[lackey context] presentRenderbuffer:GL_RENDERBUFFER];

And it only renders the boss view.

[JosephAustin]

This one's using custom render targets and i suppose it must be working, if i can just get these buffers inside the right view...

        static bgfx::FrameBufferHandle fbh = bgfx::createFrameBuffer(w, h, bgfx::TextureFormat::RGBA8);

        bgfx::reset(uint16_t(w), uint16_t(h));
        bgfx::setViewRect(0, 0, 0, uint16_t(w), uint16_t(h));
        bgfx::setViewClear(0
                           , BGFX_CLEAR_COLOR|BGFX_CLEAR_DEPTH
                           , 0xffff00ff
                           , 1.0f
                           , 0
                           );
        bgfx::setViewFrameBuffer(1, fbh);
        bgfx::setViewRect(1, 0, 0, uint16_t(w), uint16_t(h));
        bgfx::setViewClear(1
                           , BGFX_CLEAR_COLOR|BGFX_CLEAR_DEPTH
                           , 0x00ff00ff
                           , 1.0f
                           , 0
                           );

        bgfx::touch(0);
        bgfx::touch(1);
        bgfx::frame();

        [_context presentRenderbuffer:GL_RENDERBUFFER];

[JosephAustin]

I'm guessing FramebufferHandle.idx isn't the actual binding id in opengl, is it?

[bkaradzic]

Nope.

[bkaradzic]

I don't really understand what you're trying to achieve. What would be equivalent on desktop?

[JosephAustin]

I am trying to achieve a UIView which you can instance as many times as you want within an app, delete when you want, etc - like a button, a scrollview, etc. A desktop equivalent is a widget in a UI system, which a UIView actually is. I scoured the code and found that bgfx DOES directly render to the CAEAGLLayer of a UIView, so it was definitely adapted for the platform.

I don't believe anyone tested it for multiple instances. Even keeping a 'master' UIView and then using custom framebuffers to the other CAEAGLLayers (bgfx::createFrameBuffer((__bridge void*)[view layer], uint16_t(w), uint16_t(h));) does not work. It will always render directly to the CAEAGLLayers passed in as the native window handle at initialization.

I think the problem here is that iOS has to have a specific pipeline to get it to render. Your frameBuffer must connect to a renderBuffer whose storage is DIRECTLY on your target CAEAGLLayer:

 glGenRenderbuffers(1, &_colorRenderBuffer);
 glBindRenderbuffer(GL_RENDERBUFFER, _colorRenderBuffer);
 glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
                                  GL_RENDERBUFFER, _colorRenderBuffer);
 [_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(CAEAGLLayer *)self.layer];

I have to do this prior to initializing bgfx, and it doesn't work to change the buffer storage later. I tried custom render targets, but since I cannot access them directly I have no way to get them to different CAEAGLLayers.

[bkaradzic]

bgfx is more intended for cross-platform use case, than one platform specific case. If you just targeting iOS and you don't care to port your app to other platforms then you'll be much better of using Metal, since you'll have full control.

[JosephAustin]

Thats the thing, this is supposed to target ios and android, and hopefully desktop. But this is just how you do it on ios.

[bkaradzic]

Anyhow this thing above you could achieve rendering all widgets at once into one frame buffer, then using blit externally to copy data from framebuffer into separate UIViews.

[bkaradzic]

If you have custom UI system you could also just use bgfx for rendering whole UI, rather than embedding parts of it into native UI.

[bkaradzic]

Metal lacks swap chain support right now.

[JosephAustin]

Hacky solution... not sure if wise. Alternate render calls between views. At start of each render, set new platform data and re-call init. Oddly enough, this seems to work at a very good frame rate.

[JosephAustin]

I would still like to file it as a bug that views pointing at framebuffers created with different handles do not render to their respective handles in ios. It would be ideal to be able to have no 'main' nwh on this platform. This does however appear to be a workaround.

[JosephAustin]

Currently working on a fix in the render code, fyi. :)

[JosephAustin]

Well, I found out I needed to disable multithreading, but all it bought me was every view being properly cleared once. Only a view set in platformdata wants to update ever again.

It may be related to the fact that all I'm doing is bgfx::setViewClear for each id, then a touch(id), and one frame() at the end. No additional rendering yet. I know the swapchain is properly calling makeCurrent and swapBuffers.

My last thought for the night is that maybe it has something to do with all bgfx commands applying only to the first view even though I am clearly passing different viewIds. This seems likely because inserting a glClear in SwapChainGL::swapBuffers clears all the other views to the last bgfx::setViewClear setting.

[JosephAustin]

@bkaradzic - Think i found something in the documentation I'd missed that might explain my troubles. My test has been to get a bunch of views to clear to different colors, but it looks like setViewClear can only set one clear color for all the additional view framebuffers. A 'main window' can clear differently, but the last clear color set to any id over 0 applies to all of them.

There's a second setViewClear that seems to clear up to 7 different framebuffers with different colors. This suggests to me that this is expected behavior. Am I correct in this?

[bkaradzic]

It works as expected, view clear is for view, if all your views bind to exactly the same frame buffer, then last one will clear it. Indexed clear color is for MRT case, where one frame buffer have multiple attachments.

[JosephAustin]

Okay I deleted my long winded explanation from a bit ago because I think I'm better at explaining myself this way:

BGFXiOSMulti (See MyView.mm for the important parts)

Regarding your previous comment @bkaradzic , that's a bit worrying because I bind every view to a different buffer. Thus, if setViewClear IS supposed to be able to clear them to different colors, something is indeed broken. As you can see in my code sample linked here, they are pointing to different frame buffer handles with different "window handles" assigned to them. But if you run it, this is what you see:

[bkaradzic]

My suspicion is that _fbh = bgfx::createFrameBuffer((__bridge void*)_layer, w, h); is not something that works properly with https://github.com/bkaradzic/bgfx/blob/d835c09d7bfaf5e9c4d92873bfb51f3e95c232ba/src/glcontext_eagl.mm#L148

[JosephAustin]

Well, I know for a fact that it does actually bind to the correct render buffer. How I know that is that I modified that exact file you just linked and added glClear() directly after that line, and it does in fact clear the other views. It clears them to whatever the last color I named in a setViewClear call happened to be. I think if the colorRbo was failing, it wouldn't be able to draw anything at all. Could be wrong. :S

But there's also the swapchain constructor which says it should be a layer: https://github.com/bkaradzic/bgfx/blob/d835c09d7bfaf5e9c4d92873bfb51f3e95c232ba/src/glcontext_eagl.mm#L314

[JosephAustin]

https://github.com/bkaradzic/bgfx/blob/d835c09d7bfaf5e9c4d92873bfb51f3e95c232ba/src/renderer_gl.cpp#L3345

This has an interesting comment too: // iOS: need to figure out how to deal with FBO created by context.

[bkaradzic]

That's not code path for swap chain. That's only for primary back buffer created during bgfx::init.

[JosephAustin]

Ah, okay, so that's a cold trail then.

[JosephAustin]

@bkaradzic - Well I found the problem. It isn't binding to the frame buffers of the swapped views before it tries to clear.

Hack fix:

1. Expose all members of SwapChainGL globally in glcontext_eagl.h. This of course means using void* for obj-c objects, and altering the source to be function definitions.

   struct SwapChainGL
   {
       SwapChainGL(void *_context, void *_layer);
       ~SwapChainGL();
       void destroyFrameBuffers();
       void createFrameBuffers(GLint _width, GLint _height);
       void makeCurrent();
       void resize(GLint _width, GLint _height);
       void swapBuffers();
   
       void* m_context;
       void *m_layer;
       GLuint m_fbo;
       GLuint m_colorRbo;
       GLuint m_depthStencilRbo;
       GLint m_width;
       GLint m_height;
   };

2. In renderer_gl.cpp, make the correct SwapChainGL current right before clearQuad is called, like so (from line 6744)

   if (BGFX_CLEAR_NONE != (clear.m_flags & BGFX_CLEAR_MASK) )
   {
    if(view > 0) m_frameBuffers[view - 1].m_swapChain->makeCurrent();
    clearQuad(_clearQuad, viewState.m_rect, clear, resolutionHeight, _render->m_colorPalette);
   }

3. Different clear colors ahoy

This is a hack obviously. I'm sure there's some proper way to make sure the swapchain's makeCurrent function is called at the right time, but this is quite an in-depth code base ;)

[JosephAustin]

By the way, I know it should have been rendering to a back buffer there, but it does not appear to have been bound by the time that code was reached, even though it is at the top of the function. Debug mode throws an error about an invalid frame buffer.

[JosephAustin]

https://github.com/bkaradzic/bgfx/blob/b9e393e6dd126125886e02cc71ed7bc9663dd2c1/src/renderer_gl.cpp#L3229

[JosephAustin]

Better hack.

1. change renderer_gl.cpp line 3229 to m_currentFbo = m_glctx.getFbo();
2. Move definition of getFbo from glcontext_eagl.h to the .mm file

3. GLuint GlContext::getFbo()
   {
       if(NULL == m_current)
       {
           return m_fbo;
       }
       else
       {
           return m_current->m_fbo;
       }
   }