One issue is that the current UI is rendered on the CPU which is quite slow. The reason behind that is that the framebuffer has to reside on the CPU anyway due to SDL. Resolving this SDL limitation would also need to redo the UI rendering, hence these two are connected.
Overall this would save ~2ms when no UI is drawn due to removing the SDL overhead and another few ms whenever the UI is drawn.
Note: It may be possible to use SDL with OpenGL without actually including OpenGL so that should be attempted.
One issue is that the current UI is rendered on the CPU which is quite slow. The reason behind that is that the framebuffer has to reside on the CPU anyway due to SDL. Resolving this SDL limitation would also need to redo the UI rendering, hence these two are connected.
Overall this would save ~2ms when no UI is drawn due to removing the SDL overhead and another few ms whenever the UI is drawn.
Note: It may be possible to use SDL with OpenGL without actually including OpenGL so that should be attempted.