[2D MVP] Milestone 2: Improve 2d viewer performance

[sanmont3drepo]

Description

for big images the viewer seems to start dragging. this issue is meant to improve on that

Goals

• [ ] As a user I want to pan and zoom large images with a good framerate

Tasks

• [ ] Improve on the 2d viewer
• [ ] Make sure canvas resizes with onresize

[sebjf]

A brief update...

I have spent some time exploring what the browser can do with SVGs and image elements, and one approach with potential is to use the style sheet attributes to make immediate changes, while waiting for a render to occur.

From what I understand so far, there are two ways to get an SVG into a page: include its contents in the DOM, or load it in an HTMLImageElement. These approaches are entirely separated by the browser. For example, its possible to pass an HTMLImageElement to drawImage(), but not an <svg> element. Similarly, its not possible to access SVG attributes such as the viewbox when the SVG is loaded as an HTMLImageElement.

When the browser has loaded an SVG as an HTMLImageElement, it does seem to use the viewbox internally to render "properly". If a transform is applied to the Image, the SVG rasterises at the native resolution of the screen. We can also see when the viewbox is small (i.e. when zoomed far in), that the SVG rasterises (Paint event, in the browsers profiler...) quicker than when when zoomed out. This is likely because the rasteriser can cull many elements that fall outside the view.

When using an SVG inside an HTMLImageElement, updating that element's style transform will cause a re-paint. We can't control the rasterization time of an SVG, since we may get quite complex SVGs, so we need to use something as an intermediate representation.

I have an implementation that uses the style sheets to make immediate changes, then re-renders underneath and swaps the new raster when ready. This uses createImageBitamp, which is asynchronous, and transferFromImageBitmap, which should be the fastest way to blit onto a canvas. With this it can draw a 50Mb SVG with barely noticeable lag, though, since the canvas should only be blitting I would not expect any lag at all...

https://github.com/3drepo/3drepo.io/assets/5058369/849a4ff4-67cf-4e7f-83a6-a0641db9a44c

In the video above, the repainting runs every 2 seconds, so every two seconds we should see the edges get filled in. What we would also expect is that there is no noticeable stutter when the raster updates.

Personally however, I do notice a tiny stutter when it updates, and the profiler shows that createImageBitmap takes about 100 ms to complete (in the main thread), so there is clearly something still blocking. (We know this isn't the actual rendering time, because the synchronous drawImage takes about 2.2 seconds to draw the same.)

Also, a major issue is that this only works on Chrome, because createImageBitmap doesn't work properly on Firefox. The resize parameters on Firefox do not function, and these are required to get the scaling behaviour out of the vector rasteriser.

Another thing to investigate is loading the SVG as a blob in a web worker, because createImageBitmap will work in WebWorkers and allow rendering to offscreen canvases.

We could also look at, e.g. projects like https://canvg.js.org/demo/, which can run in workers (that project has a lot of outstanding issues, and the resulting rasters are not as sharp as the browsers renderer). Other Js rasterisers exist in three.js, or are cross-compiled to wasm.

[sebjf]

I have pushed a revised version of the svgImage component.

This component works by showing the SVG as an HTMLImageElement drawn on a Canvas object. When the owner sets the transform of the image, it is first applied to the Canvas itself via the style attribute. Simultaneously, a request is made to repaint the Canvas, using the same transform as is currently applied to the canvas. When the repaint finishes, the Canvas is updated, and transform used to render the SVG to it removed from its style transform.

In this way then, as the user moves around, they are moving the Canvas away from identity, and the component keeps updating the content of the canvas & moving it back, trying to keep its transform as close to identity as possible.

The Canvas is sized to be an arbitrary factor larger than the container, meaning more of the drawing is rendered than the user actually sees at a given time. When they pan, they can see some of this making the viewer appear more responsive, while waiting for the repaint to occur. This factor (overdraw) is configurable and is set to 2 currently.

There are a few conventions introduced by the component.

The two major transforms involved are Projection and T, where Projection is the transform from natural image space to Canvas space. This is fulfilled by a call to createImageBitmap. T is the transform applied through the style attribute. There is a third transform, overdrawOffset, which is a fixed offset to keep the overdrawn canvas centred in its container.

What the user actually sees on screen is a composition of Projection, T and overdrawOffset.

The region the user is intended to see is called the viewport, and should be the same size as the container. (In practice, the outer container can be a different size, viewport is mainly used when setting up the initial transform.)

When the image first loads, the component works out a transform that centres the image in the viewport and takes up as much of it as possible, while maintaining its aspect ratio. This is the origin and is also fixed.

The component has a public property, transform, which can be set externally. transform is used to move the image around relative to (origin). The setter of transform is written such that the offsets are in canvas units, and the transform-origin is the top left of the container (this is why viewport doesn't matter so much, as long as its consistent with the values used to compute origin). When transform is set, we update a member D (the desired transform) to be the composition of (Projection * T).

The component update is kicked off when transform is set. Every time this happens a counter is incremented. The redrawing happens in a callback from requestAnimationFrame. This callback re-paints the Canvas using the latest D with an asynchronous call. When the call returns the actual projection is subtracted from D, leaving a revised T to be applied via style.

Note that as redrawing may take a long time, the user may have made additional updates to transform in this time, which is why the actual projection is removed from the latest D, and we do not simply assume that T can be set to identity.

Additionally, the Projection provided to the re-painting function is in Canvas units, but under zooming, Canvas units may become smaller than the minimum size of units in natural image space. This means when the repainting is performed (which can only be specified in natural image units), the true projection is a different to that requested. Therefore, we compute the 'actual' projection expected and subtract this from D, so T contains any changes due to user actions during repainting, as well as any small offsets that cannot be expressed by the re-paint.

On each repaint conclusion, we check if transform has since been updated (via checking its counter), and if so stop there. Otherwise we call again with the latest D. This process repeats until the counter stops changing.

Though the style attribute can take a matrix, JavaScript doesn't have any native functions for dealing with matrices, so we introduce a type Transform, that consists of an offset and a scale, as well as a set of methods compose, invert, diff and multiply that act on this type using matrix conventions. These are used to do the computations on the transforms at the various stages above.

Currently, there is no resize functionality. When the drawing view is resized, it may be desirable to update the viewport and origin, as if the container becomes larger without changing the canvas size, the effective overdraw will be reduced. In theory, if origin is changed but transform is not, the user should see the view they have set up maintained, even when the canvas is re-drawn.

Additionally, this will not yet work in Firefox. For Firefox we need to replace the createImageBitmap call with a 9-parameter drawImage call, as this is the only way to get Firefox to draw an SVG at the right resolution. This latter call is unfortunately synchronous. This change is fairly straightforward, we just need to make a Promise wrapper around drawImage and change the context of the canvas.

[sanmont3drepo]

Merged into base branch