Performance ideas

[kushalkolar]

I've been trying to push this as far as possible along with @apasarkar. Some observations and ideas:

1. It seems like the framerate is much higher when using Firefox as opposed to chromium or chrome. It seemed like hardware acceleration (which I'm assuming is just was working on other sites in firefox (google maps for instance), but hardware accel wasn't working in my chromium or chrome. I wonder if that's related.
2. Is it possible to receive smaller frames and do bilinear interpolation to resize 2x or 3x before displaying in the browser, does this improve performance? Many of our datasets are images that are of low spatial resolution, but we want to look at it zoomed in on large screens.
3. Explore what's the fastest way to make the browser display images.

[almarklein]

Is it possible to receive smaller frames and do bilinear interpolation to resize 2x or 3x before displaying in the browser

Yes, I think the server could send any size in theory, using a factor of the framebuffer size is important to align with the pixel grid. Sending a smaller image should result in bilinear interpolation automatically since its an <img>.

However, I don't think it'd matter much, especially with jpeg? Well only a benchmark will tell 😉

[kushalkolar]

I just got a random idea and kinda tried it, use a webgl canvas straight within the browser to show a texture. I can get 40fps with a 1500 x 1200 size canavs and updating frames and encoding to jpeg in a slow for loop :joy: . I used this library which can create a webgl canvas in jupyter and just set textures https://github.com/K3D-tools/K3D-jupyter

@almarklein what do you think about using a wgpu or webgl canvas to display the texture in jupyter instead of image elements? Is there any way to get an image element to use webgl if that's a thing (my javascript/HTML knowledge is quite limited). If not do you think a webgl based canvas could be a part of jupyter_rfb, I can try to implement it, about time I learn some javascript :laughing:

These are the jupyter cells if anyone wants to try it out, you'll have to rotate the scene from k3d to see the texture properly:

import k3d
import numpy as np

from jupyter_rfb._jpg import encoder
from time import time

import fastplotlib as fpl
from wgpu.gui.offscreen import WgpuCanvas

canvas = WgpuCanvas(size=(1500, 1200))

plot = fpl.Plot(canvas=canvas)

plot.add_image(np.random.rand(512, 512), name="image")

plot.show()

canvas.set_logical_size(1500, 1200)
plot.auto_scale()

np.array(canvas.draw()).shape

data = encoder.encode(np.array(canvas.draw())[..., :-1], quality=80)

texture = k3d.texture(
    data,
    file_format="jpg"
)

k3dplot = k3d.plot()
k3dplot += texture

k3dplot.display()

t0 = time()
n_frames = 100

for i in range(n_frames):
    plot["image"].data = np.random.rand(512, 512)
    texture.binary = encoder.encode(np.array(canvas.draw())[..., :-1], quality=60)

print(f"fps: {n_frames / (time() - t0)}")

https://github.com/vispy/jupyter_rfb/assets/9403332/08f990f9-9164-415c-9619-8a3ddf9ceead

[kushalkolar]

I have also just gotten the latest version of chrome and it seems like jupyter_rfb is much faster in the latest version of chrome, a bit faster than firefox. I think I'm going to revisit this properly once I made a reproducible benchmark, sorry for the noise but I'm curious what you think about a webgl or wgpu canvas within jupyter :)

[almarklein]

Yes, a webgl/webgpu-based canvas seems like a reasonable idea! Interesting!

Some ideas/comments/questions that come to mind:

• You mention 40 fps, what is the fps with the <img> target and the same data?
• In the above k3dplot example, the data is encoded to jpeg, and k3dplot decodes it. I wonder how it does that, would it be an jpeg decoder in js?
• Related: One reason why the data is encoded at the server to jpeg/png is that <img> elements don't deal with raw data. If io is not much of an issue (i.e. on localhost), sending the raw data and then displaying with gpu might be a very fast path. Everything is io-bound (downloading from GPU, getting data to other process, sending to GPU again). No io cycles wasted on decoding and encoding the data. For remote sessions the cost for io would probably make this approach less fast though.
• Rendering a texture to the full viewport rectangle with webgl/webgpu should not be that much code (i.e. when we implement this, we won't need another dependency, we can use the gpu directly).