Metal Renderer: experiencing halts

[heiko-henrich]

I tried the new metal renderer. I am excited about the performance as well as with the fast adoption of this new technology into cocos2d. Thank you.

I couldn't find much about other people trying it neither in the forum nor here, so I thought you might be interested in some feedback.

There were some minor issues to get it running with my current project. e.g. I had to figure out how to compile the metal shaders and had to include the last changes of CCResponder in the MetalView.m Finally I got it running, and it is really fast. But then I experienced some halts from time to time, when about .5 seconds long, which I hadn't had so far with OpenGL.

Further investigation showed that most of the time the renderer hung in drawable = [self.metalLayer nextDrawable]; waiting for a signal. But there were also

[renderEncoder drawIndexedPrimitives:MetalDrawModes[_mode] indexCount:_count indexType:MTLIndexTypeUInt16 indexBuffer:indexBuffer indexBufferOffset:2firstIndex];indexType:MTLIndexTypeUInt16 indexBuffer:indexBuffer indexBufferOffset:2firstIndex];

and _currentCommandBuffer = [_commandQueue commandBuffer];

on top of the call stack, when I hit the stop button. There are a lot of nodes to be drawn in my project and also some background action, which didn't bother the OpenGL-renderer so far.

Maybe this is a bug in the metal framework itself (btw, I haven't tried it with 8.1 yet) Or something related just to my project (e.g. I use NSRunloopCommonModes instead of NSDefaultRunLoopMode (to use UIScrollview and also use UIKitDynamics (Yes, I know that sounds strange)). Maybe there is a possibility to fix this, but I have no idea how (maybe it has something to do with memory allocation or buffers which are too small, just a blind guess)

[slembcke]

Thanks. This is really helpful. I haven't gotten much feedback at all on the Metal renderer, and haven't gotten much of a chance to "eat our own dog food" outside of the testing app either.

The hangs you are experiencing are pretty worrisome. I'm not sure what would be causing those. Everything is still running on the main thread right? All of the methods that you listed are Metal ones. Not sure if that means it's a Metal bug or if I'm causing a race condition somehow? You aren't enabling the new iOS/GL multi-threading option are you? That hasn't been tested with Metal yet and could possibly be causing race conditions or deadlocks.

I could maybe see how the run loop mode could affect the nextDrawable, but not the other methods. Hmm. Will try it anyway.

[slembcke]

When I enabled NSRunloopCommonModes, I immediately ran into a couple of odd pauses that I was unable to reproduce ever again. Not sure if it was a red herring or not. I also left it running the sprite performance test scene which draws thousands of unbatched sprites for a few minutes. Wasn't able to get it to hang on that.

I'll try letting it run overnight on the sprite test scene and see if I can reproduce any of your issues.

[heiko-henrich]

Hey Scott.

Thank you for working on this. Meanwhile, I looked from the other side and enabled NSDefaultRunloopMode (and removing the scrollview): The stuttering got even worse, so the problem shoulb be somewhere else. I disabled threaded rendering (btw worked really great with GL) I couldn't even get the app started with it.

I tried to increase the vertesand uniform capacity: also no effect.

When halting, [self.metalLayer nextDrawable] returned nil, so I had to activate the commented code.

Meanwhile I updated to iOS 8.1, so if it is a bug in the framework, it didn't get fixed.

There is really quite a lot going on in my app, Several thousend nodes, some including textures and also several background threads. It is probably an extreme case.

It looks somehow like java garbage collection used to halt the entire program for a few seconds. So maybe it's just too much for the metal renderer to cope with. But then again, how will they cope with big desktop games, they're advertising with.

Did you have any luck with your stress tests?

Heiko

Am 20.10.2014 um 22:07 schrieb Scott Lembcke notifications@github.com:

When I enabled NSRunloopCommonModes, I immediately ran into a couple of odd pauses that I was unable to reproduce ever again. Not sure if it was a red herring or not. I also left it running the sprite performance test scene which draws thousands of unbatched sprites for a few minutes. Wasn't able to get it to hang on that.

I'll try letting it run overnight on the sprite test scene and see if I can reproduce any of your issues.

— Reply to this email directly or view it on GitHub.

[slembcke]

When halting, [self.metalLayer nextDrawable] returned nil

Grr. I thought that was a bug they fixed. It used to happen to me constantly (like every 30 seconds or so). I could swear there was a note about it being fixed in some of the change logs... Will have to dig for it. Sounds like I have to put the workaround back in. -_-

I left the demo app running on my iPad for a few hours but it never got stuck. It sounds like you are really stressing the performance, so what is your framerate like? I'm wondering if I need to throw more work at the GPU or add artificial delays. Perhaps it's a timing issue.

[slembcke]

Well. Turns out there was a discussion thread on Apple's developer forums about the nextDrawable issue where they shared some insider knowledge. The docs say that nil is only returned if the CAMetalLayer is set up improperly. According to the thread, the layer has a limited pool of drawables and the nextDrawable will block with a short timeout value.

This leads me to think that it's returning nil because your frames are taking too long to render and the CPU is getting ahead of the GPU. Should be easy to try at least. There is a semaphore in the CCMetalView class that prevents buffering too many frames. I can probably use that for the drawables as well if I move some calls around. Maaaaybe fixing that would prevent the other pauses? (optimistic thinking)

[slembcke]

Ok! I think I reproduced the issue, and have something to investigate. I made a scene that drew a few hundred fullscreen sprites to bog down the GPU. It then started returning nil drawables. Even after exiting that scene, the rest of the app seemed to perform badly, though I can only guess why at this point.

Next I was going to see if the semaphore could be used to manage the drawables as well. It turns out that the call to dispatch_semaphorewait() had been completely removed at some point. -- That explains a lot actually. It would only be a problem if the performance got too bad and it was trying to buffer too many frames at once.

With the semaphore fixed so that it properly limits frame buffering, it seems to work well again after a few minutes of testing. I'm still not convinced I reproduced the same issue as you however. I pushed the fix to develop if you want to give it a try.

edit: Nope. The pauses are still happening. Drat.

[heiko-henrich]

Hey Scott.

This sounds like good news!!! I wondered about this lonely signaling semaphore and thought it was something like an advanced ARC trick... In my project, the nodes are overlapping, fairly large, though just color nodes, so your discovery makes sense for me. A kind of frame buffer overflow?!?!?!? Interestingly the performance is still at 60fps. I even threw out the prerendered label textures and the updating of the infinite scrolling, and still seeing the hangs. I will try the fix as soon as possible. Because I`ll go on a trip until mid of next week, I can't promise to do it before.

Btw, thank you for the great and professional work you're doing. It's a pleasure to see cocos2D evolve this way.

Heiko

Am 21.10.2014 um 19:14 schrieb Scott Lembcke notifications@github.com:

Ok! I think I reproduced the issue, and have something to investigate. I made a scene that drew a few hundred fullscreen sprites to bog down the GPU. It then started returning nil drawables. Even after exiting that scene, the rest of the app seemed to perform badly, though I can only guess why at this point.

Next I was going to see if the semaphore could be used to manage the drawables as well. It turns out that the call to dispatch_semaphorewait() had been completely removed at some point. -- That explains a lot actually. It would only be a problem if the performance got too bad and it was trying to buffer too many frames at once.

With the semaphore fixed so that it properly limits frame buffering, it seems to work well again after a few minutes of testing. I'm still not convinced I reproduced the same issue as you however. I pushed the fix to develop if you want to give it a try.

— Reply to this email directly or view it on GitHub.

[heiko-henrich]

Bad news. Just tried it with the stripped down version of my project. Still halting. At least there are no nil drawables any more. I tried to reduce the queued frames to 1, no effect (albeit a lower frame rate) Maybe it's just(?) an Apple bug after all.

Heiko

Am 21.10.2014 um 19:14 schrieb Scott Lembcke notifications@github.com:

Ok! I think I reproduced the issue, and have something to investigate. I made a scene that drew a few hundred fullscreen sprites to bog down the GPU. It then started returning nil drawables. Even after exiting that scene, the rest of the app seemed to perform badly, though I can only guess why at this point.

Next I was going to see if the semaphore could be used to manage the drawables as well. It turns out that the call to dispatch_semaphorewait() had been completely removed at some point. -- That explains a lot actually. It would only be a problem if the performance got too bad and it was trying to buffer too many frames at once.

With the semaphore fixed so that it properly limits frame buffering, it seems to work well again after a few minutes of testing. I'm still not convinced I reproduced the same issue as you however. I pushed the fix to develop if you want to give it a try.

— Reply to this email directly or view it on GitHub.

[slembcke]

To be clear, when you say "halting" do you mean it gets stuck permanently or that it pauses for a fraction of a second and continues?

I think I have another lead I can track down for the pauses. I was actually able to catch one of the pauses in the time profiler instrument with the sample waiting threads option enabled. The main thread was put to sleep for hundreds of milliseconds waiting to create a texture. I can ask around on the Apple forums if there is a way to avoid that.

Are you creating a lot of textures on the fly? (CCLabelTTF, textures or sprites from CGImages, etc) It might be triggered by other resource allocations as well like resizing buffers, but it's pretty difficult to know when that's happening.

[heiko-henrich]

To be clear, when you say "halting" do you mean it gets stuck permanently or that it pauses for a fraction of a second and continues?

Not permanently, so let's say 0.5 secs or so. Are you creating a lot of textures on the fly? (CCLabelTTF, textures or sprites from CGImages, etc)

Yes I do that, but I reduced this to almost nothing in the stripped down version. I'll try to switch this off entirely.

It might be triggered by other resource allocations as well like resizing buffers, but it's pretty difficult to know when that's happening

One thing I noticed was, that there were these halts when a lot of CCColorNodes resized simultanuesly. Does this trigger some kind of buffer resizing?

[slembcke]

So it does seem to be resource creation. I put logging around creating buffers and textures and got this:

2014-10-22 10:59:13.772 cocos2d-tests-ios[1646:331332] Buffer creation took 1017.224958 ms. 2014-10-22 10:59:38.256 cocos2d-tests-ios[1646:331332] Sampler/texture creation took 20.847083 ms. 2014-10-22 10:59:39.258 cocos2d-tests-ios[1646:331332] Sampler/texture creation took 994.352917 ms. 2014-10-22 11:00:11.939 cocos2d-tests-ios[1646:331332] Render pipeline state creation took 4.768042 ms. 2014-10-22 11:02:46.857 cocos2d-tests-ios[1646:331332] Sampler/texture creation took 997.387042 ms.

Buffers creation is sort of complicated and a little random. Buffers are used by renderers. Buffers are pooled. Renderers are mostly created up front, but using render textures might exhaust the pool and create more. A renderer may also resize it's buffers if too much geometry is drawn to fit in its existing buffers. So... really hard to say what causes buffers to be created.

I'm not sure what else to look for on my end. Hopefully we get a helpful reply on the Apple Metal forums.

[heiko-henrich]

Is there a way, to get the Metal renderer out of the main thread? I realized, that there is also this UIDynamicAnimator running, which is (of course) also running on the main thread and (probably) using the display link interrupt. This could also allow pre rendering of CGImages or TTF textures, without disturbing the renderer.

Am 22.10.2014 um 20:20 schrieb Scott Lembcke notifications@github.com:

So it does seem to be resource creation. I put logging around creating buffers and textures and got this:

2014-10-22 10:59:13.772 cocos2d-tests-ios[1646:331332] Buffer creation took 1017.224958 ms. 2014-10-22 10:59:38.256 cocos2d-tests-ios[1646:331332] Sampler/texture creation took 20.847083 ms. 2014-10-22 10:59:39.258 cocos2d-tests-ios[1646:331332] Sampler/texture creation took 994.352917 ms. 2014-10-22 11:00:11.939 cocos2d-tests-ios[1646:331332] Render pipeline state creation took 4.768042 ms. 2014-10-22 11:02:46.857 cocos2d-tests-ios[1646:331332] Sampler/texture creation took 997.387042 ms.

Buffers creation is sort of complicated and a little random. Buffers are used by renderers. Buffers are pooled. Renderers are mostly created up front, but using render textures might exhaust the pool and create more. A renderer may also resize it's buffers if too much geometry is drawn to fit in its existing buffers. So... really hard to say what causes buffers to be created.

I'm not sure what else to look for on my end. Hopefully we get a helpful reply on the Apple Metal forums.

— Reply to this email directly or view it on GitHub https://github.com/cocos2d/cocos2d-swift/issues/1034#issuecomment-60130655.

[slembcke]

Oh, sorry. I missed your last question.

Do you adding Metal support for CC_DIRECTOR_IOS_THREADED_RENDERING or running a separate private Metal renderer on a separate thread?

The former is on my list. The very first project I tried making with 3.3 I wanted to use the threaded renderer on older hardware AND the Metal renderer for fancy effects on new hardware. It never occurred to me that you can't do both in the same binary. -_- It shouldn't be hard, it's just not a high priority yet.

[davidtsai]

I am experiencing this same issue. The repro I have right now is that sometimes it pauses when new CCLabelTTFs are created, but it only happens occasionally. However, I have also seen it pause on its own when more sprites were being rendered. Everything runs smoothly with openGL.

[davidtsai]

Think I have a fix for this from examining the example Metal project: https://developer.apple.com/library/prerelease/ios/samplecode/MetalBasic3D/Introduction/Intro.html

Everything is significantly faster now. Pull request above!

[heiko-henrich]

Great. It works finally. Thank you.