This repository is for the legacy Mixed Reality Toolkit (MRTK) v2. For the latest version of the MRTK please visit https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity
I was tracking a warning in Keyboard.cs that is a potential bug when dictating text. On this line the inserted text does not actually get assigned. To test this I ran the KeyboardTest scene on the HoloLens and noticed that the performance is really bad. I could barely use the Keyboard as it was lagging so much. If you want to test this, make sure to add the keyboard to one of the input fields in the test scene because otherwise, it will just throw errors (also disable the slider) http://i.imgur.com/Tx7jfQj.png.
I've added the FPSDisplay to the scene and recorded a short clip showing this. Note that the FPS drops to about 19 due to the recording: https://puu.sh/x97jB/6367b1a46c.mp4. The FPS then proceeds to drop to 7 while looking at the keyboard.
I kinda went down the rabbit hole and started investigating this issue within the editor. The problem I found has to do with how Unity draws UI in batches. Similiar objects on the same canvas can be drawn at the same time when certain conditions are met. These conditions are NOT met with the keyboard and the result is that each individual key is drawn separately like so in the frame debugger: https://puu.sh/x95Zp/03ed266e3b.webm. You can see the batch count at the top right from the scene stats goes up to 96 and I've seen it go up as high as 98 when moving the cursor.
1. Z-Positions
The first thing that breaks batching is different z-positions. All the main sections of the Keyboard have a slight z-offset which totally destroys batching. Setting all the sections http://i.imgur.com/btSdqkA.png to a z-position of 0 reduced the batches to 21. This has solved most of the problem, however we can do better.
2. Sprite Atlas
Looking at the new UI profiler one can see that a lot of batches are broken due to different textures on the elements like the back button, shift keys, button highlights and pretty much all images and textures that are used here. Unity 2017 has a pretty good new feature for this called a sprite atlas. It builds a single sprite out of all the added ones to be able to make them similar. This has to be enabled in Editor settings > Sprite Packer > Always enabled. Creating a new atlas somewhere under HoloToolkit/UI and adding both the Images and Textures folders in there gets it down to 15http://i.imgur.com/zG5DlKx.png.
We are getting an additional batch from the InputFieldBackground on the input field on the keyboard. I couldn't figure out how to get the actual sprite for that so I can add it to the atlas so I just removed that sprite from the input field for now, which puts us at 14.
3. Hierarchy Order
The last one took me a while to figure out. In the profiler, it's shown that the currently highlighted keyboard key has its own batch. It turns out that the order of elements can be an issue too. The hierarchy for most of the keys looks like this:
X_Button
Highlight
keyboard_X
Swapping the order of the 2 child elements will add the keyboard key to the batch with all the other ones. This reduced the batches down to 13, 3 of which are actually from the Keyboard http://i.imgur.com/nXfNh39.png. The rest of the batches come from other elements in the scene.
The frame rate still drops when not recording when using the keyboard but it's not near as bad as before. Another issue I think is that there are a lot of unnecessary ray cast targets that we don't require. Like all the highlight and text elements are ray cast targets when just the parent would suffice. The keyboard is a world canvas and has an attribute for an event camera. Not setting this will call Camera.main for every ray cast.
I'm not sure if the issue with the gaze manager and UI elements with the same z-position still exists (#620). If it does this probably has to be fixed first so that the keyboard can be rebuilt like this.
StephenHodgson
commented
7 years ago
I was tracking a warning in
Keyboard.csthat is a potential bug when dictating text. On this line the inserted text does not actually get assigned. To test this I ran theKeyboardTestscene on the HoloLens and noticed that the performance is really bad. I could barely use the Keyboard as it was lagging so much. If you want to test this, make sure to add the keyboard to one of the input fields in the test scene because otherwise, it will just throw errors (also disable the slider) http://i.imgur.com/Tx7jfQj.png.I've added the FPSDisplay to the scene and recorded a short clip showing this. Note that the FPS drops to about 19 due to the recording: https://puu.sh/x97jB/6367b1a46c.mp4. The FPS then proceeds to drop to 7 while looking at the keyboard.
I kinda went down the rabbit hole and started investigating this issue within the editor. The problem I found has to do with how Unity draws UI in batches. Similiar objects on the same canvas can be drawn at the same time when certain conditions are met. These conditions are NOT met with the keyboard and the result is that each individual key is drawn separately like so in the frame debugger: https://puu.sh/x95Zp/03ed266e3b.webm. You can see the batch count at the top right from the scene stats goes up to 96 and I've seen it go up as high as 98 when moving the cursor.
1. Z-Positions
The first thing that breaks batching is different z-positions. All the main sections of the Keyboard have a slight z-offset which totally destroys batching. Setting all the sections http://i.imgur.com/btSdqkA.png to a z-position of 0 reduced the batches to 21. This has solved most of the problem, however we can do better.
2. Sprite Atlas
Looking at the new UI profiler one can see that a lot of batches are broken due to different textures on the elements like the back button, shift keys, button highlights and pretty much all images and textures that are used here. Unity 2017 has a pretty good new feature for this called a sprite atlas. It builds a single sprite out of all the added ones to be able to make them similar. This has to be enabled in Editor settings > Sprite Packer > Always enabled. Creating a new atlas somewhere under HoloToolkit/UI and adding both the Images and Textures folders in there gets it down to 15 http://i.imgur.com/zG5DlKx.png. We are getting an additional batch from the
InputFieldBackgroundon the input field on the keyboard. I couldn't figure out how to get the actual sprite for that so I can add it to the atlas so I just removed that sprite from the input field for now, which puts us at 14.3. Hierarchy Order
The last one took me a while to figure out. In the profiler, it's shown that the currently highlighted keyboard key has its own batch. It turns out that the order of elements can be an issue too. The hierarchy for most of the keys looks like this:
X_Button
Highlight
keyboard_X
Swapping the order of the 2 child elements will add the keyboard key to the batch with all the other ones. This reduced the batches down to 13, 3 of which are actually from the Keyboard http://i.imgur.com/nXfNh39.png. The rest of the batches come from other elements in the scene.
The Result
Drawing the keyboard has been sped up tremendously as you can see in the frame debugger: https://puu.sh/x95W2/3d297b03cf.webm. And the FPS does not drop into oblivion anymore on the HoloLens https://puu.sh/x99pp/9f85ad804b.mp4.
The frame rate still drops when not recording when using the keyboard but it's not near as bad as before. Another issue I think is that there are a lot of unnecessary ray cast targets that we don't require. Like all the highlight and text elements are ray cast targets when just the parent would suffice. The keyboard is a world canvas and has an attribute for an event camera. Not setting this will call
Camera.mainfor every ray cast.I'm not sure if the issue with the gaze manager and UI elements with the same z-position still exists (#620). If it does this probably has to be fixed first so that the keyboard can be rebuilt like this.
Resources
Most of this I've gathered from the 2 recent unity optimization talks which are really great to listen to. https://www.youtube.com/watch?v=eH-PdFKgctE https://www.youtube.com/watch?v=_wxitgdx-UI
I've got a branch over at https://github.com/Zod-/MixedRealityToolkit-Unity/tree/Keyboard_Optimization with these changes in the
KeyboardTestscene.