Feature request - Object caching

[gordyr]

Fabric.js is in my opinion very well written with excellent overall performance.

However there are occasions where the performance is drastically reduced due to performance limitations of the html5 canvas API and not down to fabric itself.

Two of these occasions are:

• When manipulating a text object using a very complex font or with a textShadow applied
• When manipulating objects with shadows turned on (particularly with a large blur radius)

Having gotten quite familiar with the source code over the last few weeks I can see a very clear way of rectifying these performance issues while dramatically increasing overall manipulation performance across the board.

My proposal is to cache objects (on demand, not continuously) as off screen canvas's so that during manipulations (moving, rotating etc) the object is drawn to the canvas using drawImage() rather then being completely redrawn.

The flow as I see it would go as follows:

• when mousedown occurs on an object an offscreen canvas is temporarily created
• the temporary canvas is set to the size and width of the object and the object is rendered to this canvas instead of the main lower one
• during manipulations the the _render() is halted and drawImage(), using the temporary canvas as the source element, is called instead.
• on mouseup the temporary canvas is nulled to avoid creating excessive in memory canvases.

I have been hacking away at the fabric source code and have implemented this myself. For text objects that use extremely complex fonts or ones with text shadows etc, the performance increase is from about 10fps up to 60fps. The same goes for image objects with shadows set. The visual result is identical to rendering the objects each frame, just much faster.

Everything works beautifully for moving and rotating however my implementation currently leaves an issue with scaling. Since we are dealing with a canvas element the quality changes as you scale, then on mouseup is re-rendered correctly. This is mostly only an issue when scaling up, but for text it is also heavily noticable when scaling down.

In order to get around this some sort of throttle or timer could be used to rebuild the cached canvas at the new size in the background. Hopefully leaving the visual difference almost imperceptible. So far I haven't managed to get around to attacking this yet.

Implementing the above is actually relatively simple, but with my lack of knowledge of Fabric's inner workings (particularly around the scaling/rendering) means that my code is perhaps more complex than it needs to be, hence my leaving it as a feature request rather than making a pull request.

Also... Another huge benefit of using the above method mean it solves a large bug in Chrome on Windows....

Currently in Chrome on windows when rotating a text object the letter jiggle continuously. I assume this is down to Chrome not properly implementing sub-pixel rendering on canvas text objects. Using the above caching method this jiggling completely disappears.

I would be interested to hear your thoughts Kangax.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[gordyr]

For what it's worth I have just implemented a scaling timer as mentioned above and the results are excellent.

I have also taken it one step further by also caching the rest of the canvas (aside from the object being manipulated) when an object starts to be manipulated.

the flow now looks like this:

• when MOUSEMOVE occurs on an object that is being transformed, an offscreen canvas is temporarily created for the selected object AND the rest of the canvas is rendered once to a second offscreen canvas.
• during manipulations we simple perform two drawImage calls, one for the complete canvas containing all non moving objects and the second for the object being manipulated.
• on mouseup both the temporary object canvas and the main background canvas are nulled. We then perform a final renderAll() call to reset everything.

In terms of performance, its fantastic. I've tested by using a loop to create 1,000 objects all with shadows and the object manipulation performance is just as fast as with one object since the rendering pathway is never more than two drawImage() calls plus the controls.

I would imagine if we continued to scale this up there would be a point where the time taken to render once (before both the background and manipulation object are cached) would become noticeable on mousedown. In which case we would need to look at potentially precaching, perhaps on hover to alleviate this.

There are however drawbacks to this approach:

• When using background caching as well as object caching, the moving/scaling/rotating object will always be 'on top' of the other objects. For my purposes this is perfect and exactly what I want. However this might not be right for all use cases.
• When manipulating a text object the text is ever so slightly blurrier (only while mouse is down) than drawing the whole thing each frame. I believe this is down to how sub-pixel rendering is handled differently between drawImage() and fillText(). The difference is very minor and not at all distracting, however this may not be appropriate for some.

For this reason it feels as though it would be best to offer both object caching and background caching as separate options with object caching being a switch per object and background caching being a global option.

I haven't yet attacked group selection but this should essentially be exactly the same process except we will be caching a whole group instead.

[kangax]

Hey @gordyr, thanks for tackling this! (and sorry for late reply)

As you can see in issue 318, we had similar functionality (stubbing) at one point but eventually dropped it as it wasn't complete; mainly due to special cases that you mention, like object or group scaling.

However, now Fabric supports static canvas. And there's also a lot of people who don't need interactivity, and so could use very simple caching mechanism (with manual invalidation) without any problems. That's why I wanted to introduce at least manual caching for cases like that, and leave dynamic discarding, etc. for later.

In order to get around this some sort of throttle or timer could be used to rebuild the cached canvas at the new size in the background. Hopefully leaving the visual difference almost imperceptible. So far I haven't managed to get around to attacking this yet.

Hm, I'm generally not a big fan of timers, as they tend to affect performance. But it's an option. The way I approached it back in the days was to discard cached image as soon as shape is being scaled. Then generate a new cached one once scaling is complete. That way, you always scale vector shape without any quality loss, albeit at a slower speed.

Currently in Chrome on windows when rotating a text object the letter jiggle continuously. I assume this is down to Chrome not properly implementing sub-pixel rendering on canvas text objects. Using the above caching method this jiggling completely disappears.

I noticed something like this too. Glad this fixes it.

For what it's worth I have just implemented a scaling timer as mentioned above and the results are excellent.

Nice! I'd love to see it :)

When using background caching as well as object caching, the moving/scaling/rotating object will always be 'on top' of the other objects. For my purposes this is perfect and exactly what I want. However this might not be right for all use cases.

Yeah, it definitely complicates things.

When manipulating a text object the text is ever so slightly blurrier (only while mouse is down) than drawing the whole thing each frame. I believe this is down to how sub-pixel rendering is handled differently between drawImage() and fillText(). The difference is very minor and not at all distracting, however this may not be appropriate for some.

One option could be to render text at a slightly higher scale, for cached image. Just a thought.

For this reason it feels as though it would be best to offer both object caching and background caching as separate options with object caching being a switch per object and background caching being a global option.

I actually never even thought about caching background. But this goes in line with dirty rectangles optimization which I've been meaning to implement as well at some point (basically only render active object and any objects that it intersects with).

In any case, I'd love to see what you have so far. And thanks again.

[gordyr]

Just thought I'd give you an update on this so far...

Firstly, following some more detailed testing I have decided on a quite different flow. Originally all objects were only being cached once manipulations were occurring on an active object, they were then being destroyed on mouseup. My intention was to minimize memory allocation. While this worked very well for the most part, once we started hitting a huge number of objects 1000+, or when we had a fair number of extremely complex objects with expensive operations like shadows etc, we were experience a perceptible delay when building the cache for the active object. It was very minor, but enough to fire a alarm bells in my increasingly OCDish mind.

My objective has been to make all manipulations from mousedown to mouseup run at 60fps regardless of the number of objects being rendered and regardless of their complexity.

This objective is now achieved.

I did some more in depth memory testing and found that holding these off screen canvases for each object was far less expensive in terms of memory allocation that I initially expected. For that reason my current implementation now automatically creates a cached version of an object whenever _onObjectAdded() is fired.

Then within renderAll() and _draw() it is only ever the cached version that is drawn to the screen. All other rendering methods simply draw and update the cache. This way we are always seeing the cached versions of the objects which removes the slightly perceptible differences between native text rendering and the cached drawImage rendering.

Three new properties are added to the Fabric canvas initialization which currently default to the following:

cacheobects : false, cachebackground : false, oversample : 1.5

the oversample property is the factor by which we oversample the cached objects. This is only applicable when cacheobjects = true.

The oversample property came about from your suggestion to render text at a slightly higher resolution and to resize the cached image down when drawing it. The default setting of 1.5 makes the result look perfect to eyes. Text is just as sharp as when rendering natively. I have however left this exposed so that we can play with this setting should we need.

Just to explain how the rendering pathway now works in this implementation:

cachedobject : true / cachedbackbackground : false

RenderAll() draws each and every cached object, from its cache. The end result provides exactly the same selection and manipulation behaviour as standard non-cached Fabric.js

cachedobject : true / cachedbackbackground : true

With cachedbackground set to true, we take the optimization a step further. A new canvas layer is added behind the lower/upper canvases, this is physically added to the DOM.

When an object becomes active this cachedbackground element is updated. All objects apart from the active one (or active group) are drawn to the background layer from their cached representations to keep this fast.

This way whenever we are performing any kind of manipulation on an object or group (yes it works perfectly with grouped objects) all we are doing is performing a single drawImage() call each frame.

The caveat of this method is that while objects are being manipulated they will always be on top, returning to their correct stack locations once the manipulation is finished (on mouseup)

Whenever an object needs updating we now call canvas.renderObject(object); which will update the cache of the specific object and redraw where needed:

obj.set('color', '#ffa903');
canvas.renderObject(obj);

This essentially replaces renderAll() when using caching. This call can of course be added to the .set() method, however as you have rightfully stated elsewhere, many people probably still use standard object dot notation in their code. For now I have left it out and simply replaced RenderAll with renderObject in my application code. This kind of makes sense to me as we are no longer rendering everything all the time. And when using caching, it would be wise to encourage people to think in terms of rendering objects/groups independently.

However... One of my next steps is to implement automatic cache invalidation so that this method becomes unneeded and my caching code can simply be a transparent addition to Fabric should you deem it worthwhile.

Other points

The reason for using a timer during scaling operations is because trying to rebuild the cache of a complex object or one with large shadows, makes the manipulation performance far slower, as once again we are redrawing the object everytime mousemove is fired. Although the performance is still much better than the non-cached method even without the timer it was not good enough for my liking. By using a timer we are essentially throttling the complete redraw and instead rendering a scaled version of the cache inbetween these complete redraws.

The timer is began as soon as an active object begins a scaling operation and is stopped as soon the scaling ends on mouseup. I will be switching the interval out for requestAnimationFrame to further optimize this and to maximize complete redraw frequency dependent upon computer performance.

There are a couple of other convenience methods I have added to enable the user to flag an object as requiring constant cache updates.

These are currently useful in apps such as mine.

e.g. when using a colour picker or slider to change the colour of text we set the objects ._changing flag to true.on slidestart. Then while the colourpicker is being manipulated the colour of the text updates continuously, since the cached image is being rebuilt on every call. On slideend, the _changing flag is returned to false.

Once automatic cache invalidation is in this will also be unneeded.

One of the best aspects of all this is, other than the dramatic performance improvement) is that text rendering is now consistent between browsers. I touched on the jiggling text in windows Chrome when rotating a text object. Since everything is now being rendered from cached images this is of course eradicated, not only that but since Chrome performs some sort of antialiasing on drawImage calls it now appears to match Firefox and IE, almost exactly.

I haven't created a pull request yet as there is still further work and testing required as I haven't gotten around to trying SVG's yet. But in terms of text and image objects, the solution appears flawless.

I am developing this in the background alongside my app, so cannot focus purely on Fabric. The functionality is being added/modified as I need it. SVG's are part of my app design so I would expect to be implementing and testing it thoroughly some time this week.

I might knock up a quick video to demonstrate the performance differences if I get a chance....

Thats about it for now. i'll get back to you soon.

[asturur]

duplicate of #318