Alternative Rendering

[Updownquark]

I'm not sure how this could be done, but it would be ideal if it was possible to plug in different renderers than just the plain java Graphics implementation. Examples would be JavaFX and OpenGL. These would make MUIS much more performant.

[Updownquark]

I think instead of having pluggable or switchable rendering, that I'll eventually just switch over to OpenGL completely. The only possible problem with this may be security. If I give a library an OpenGL "session" or whetever, how much power does it have through that? Is it governed by permissions? Can they take over the video card or crash it or anything? Can I gain any control over how much GPU resources a thread takes? Need to investigate.

[Updownquark]

I think I'll include another rendering architecture improvement here. The re-rendering should be done to a buffered image, which is then rendered all at once to the display. Would look much better.

We also need to think about double-buffering some time. Probably not all elements would use it, but it would be a good option to have for some.

[Updownquark]

Having a problem with rendering on issue #2 (Border Layout). Not at all sure that this is the only issue, but one thing that is an issue is that the rendering is taking place in 2 threads--the AWT Event Queue and the MUIS Event Queue. Rendering in AWT happens on initial rendering and due to resizes of the browser window. Rendering on the MUIS threads happens for those things and in response to MUIS events, like state changes causing a different style to become active.

The best way I can think of to overcome this problem is to do what I said above with the buffered image. I need to redo the architecture by which MUIS draws things on demand. Here's how it will work.

• There will be a buffered image or some structure containing one in MuisDocument.
• When the AWT thread requests a redraw, this image is drawn without consulting any other MUIS objects. If this doesn't exist (because the first draw has not completed, for example), then nothing is drawn.
• When the image becomes out of date due to some event (user event, animation time change, etc.), a new image will either be created from scratch and rendered completely, or cloned from the old image and a piece of it updated. The new image will then replace the old one in MuisDocument and this will be drawn to the source graphics. This process should only ever happen on the MUIS event thread.
• Rendering events should be batched. Ideally, isolated events should only result in a single new image. In the case of something that requires frequent rendering updates, however, the render should happen on a configurable interval--a frequent one so animation looks good.
• The structure should contain metadata about the bounds of the elements that were rendered to it. Since this is what the user sees, positioned user events need to propagate to elements according to the structure that was rendered at the time when the event occurred. This is done now, but the information is stored in the elements themselves. It would make sense to move it to the structure in MuisDocument for single-source.

[Updownquark]

I wanted to write that down and finish my thought, but as I was making that list, I was realizing that it's very possible that none of that will make any sense once I switch over to OpenGL. Even in AWT land animation performance might be terrible with that architecture. Might have to work around it somehow. Kinda looking like I might need to work the OpenGL part of this issue before I do anything about buffers

While I'm doing the brain dump thing, an idea that just occurred to me is that maybe the animation needs to coordinate more with the event queue. Maybe instead of having its own thread, the motion package should just fire events on the MUIS event thread. This would allow the renders to happen quicker (don't have to wait for another thread).

[Updownquark]

Waffling on this a little bit. If I can get the performance to be good without OpenGL, there would be some benefits to that:

• Pure Java. No native code required for the core.
• Headless compatible. Most toolkits would work equally well in headless environments.
• Security. I'm not sure if OpenGL would interfere with my ideas for security in MUIS (talked about this a little bit above).

Additionally, even if I do end up going OpenGL, a lot of the architecture changes I would do (mentioned above) would still be useful. I think I'll do that and maybe some profiling and optimization and see if I can get things to perform more reasonably.

[Updownquark]

I think I'll call this done except for the OpenGL part, which I may or may not ever do. That will depend on whether I can get performance to be good for more complex documents, which I have yet to write. The only part of the list above I didn't do is the render event batching, but I did batch the resize events, which helped a lot. Didn't even have to modify the eventing architecture for that. I can batch the paint events later if I decide there's enough benefit to it.

I'll close this for now and reopen if and when I decide to tackle OpenGL.

[Updownquark]

I'm reopening this because the new architecture changes caused delays in repaints. Need to fix this.

[Updownquark]

Fixed this. There is, however, still something of an issue. When a widget is rendered now, it's rendered to both the stored image in the document and to the actual device graphics. This makes things more responsive than before, but since the rendering is to the device, the progress of rendering operations that take significant time can be watched on the screen, which doesn't look very nice.

[Updownquark]

Still have delays between clicking on an element and the stateful style color displaying on the screen

[Updownquark]

Although the delay is now gone, I still want to do another optimization. Even though evaluations are now pretty fast, there's absolutely no need to do as many as are being done for each state change. I want to change the style sheet to return a subclass of StyleExpressionValue whose getValue() method evaluates the value. Thus the expressions in a style sheet can be investigated without any unnecessary evaluations.

[Updownquark]

As noted above, the fix was actually in PRISMS. In detail, the slowdown was due to the calling (thousands of times each) of a utility method to determine whether an package-like expression was indeed a package on the classpath, and Class.forName. I delegated the Class.forName call to a new utility method in the same class as the package tester method and added caches for each. This cut the state change time for the "pressed" state down from about 1/3 second to 1/50 on my home machine.

Then, after adding the feature described above for evaluating animated style sheets only as needed, the time went down to about 1/200 of a second. It's possible other performance issues related to styles will pop up, but I made them a ton better. This is done for now.