Performance tests

[julenka]

Describe the problem

We sometimes receive feedback that MRTK causes apps to run slowly on HoloLens. For example, last week a colleague brought up a customer whose app was running at below 60 fps. Sometimes these issues are due to MRTK. But many times, the issues are because of mis-configuration of the applications, not because of bugs in MRTK. Wouldn’t it be nice if we could have data to confidently say, “no MRTK is not causing this regression”?

One way we could do this is with a set of performance tests. These are tests that would verify that MRTK continues to run at 60 fps under typical and atypical conditions. We could use these as evidence that MRTK is not the performance bottleneck, and to ensure that new changes do not regress performance.

Describe the solution you'd like

Add a new class PerformanceTests to the playmode test suite that does the following:

1. Plays a scene with default MRTK configuration
2. Brings up 1, 2 hands
3. Moves the hands around
4. Every frame, ensures that the frame takes no longer than 5 ms to run.

We would do steps 1 – 4 for the following scenarios:

• [ ] The HandInteractionExamples scene
• [ ] A scene with 1,000 grabbable colliders
• [ ] A scene with 1,000 touchable colliders
• [ ] A scene with 1,000 colliders that are neither touchable nor grabbable
• [ ] A scene with 1,000 pressable buttons and a spatial mesh

... please add any other cases which might cause performance to regress. I am most familiar with issues caused by the input system.

Describe alternatives you've considered

An alternative I considered was just to direct people to documentation and provide tools to improve performance. MRTK already provides guidelines for improving performance as well as a tool for optimizing scenes for AR headsets

The performance tests would supplement these by providing data that MRTK is not slowing down performance, and also would ensure that people do not check in code which regresses performance.

[darax]

I would recommend identifying a representative load of features (mesh on, 13 buttons in the scene, 3 solver things, etc), set a CPU and memory budget, and measure against that budget. Perhaps there could be multiple loads and we could document the cost of each load.

To measure, I'd use the performance tracing in the Device Portal website. The problem with just making sure you keep 60FPS is that there's a cliff where you can start really far way (processing would allow for 180hz) and slowly creep toward the cliff. For a framework, like the MRTK, we should leave as much headroom as possible to the application.

[IssueSyncBot]

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!