Increasing framerate

[bordaigorl]

Here's an idea. In theory, using javascript and a canvas element one could render the raw data stream in the browser. This would make the proxy mode redundant and give flexibility: for example one could try to send only updated parts with coordinates and the js would only redraw the interested region of the canvas. Device events could be used to identify the affected region (i.e. the one around the pen events) and for good measure a whole frame can be sent at regular intervals.

I have no idea if this attacks the real bottlenecks.

[Merovius]

For a USB-connection, this would probably help. For a wifi-connection, AFAICT, the network becomes the bottle-neck without compression (and the CPU or context-switches or something becomes the bottleneck with). (edit: Of course, the upside of not needing a proxy still applies)

I have thought about this as well, and it might still help because we might be able to use a simpler compression or at least have more control over it than with PNG. I'm also not sure how much javascript I want to add and maintain - a full image compressor seems overkill to maintain ourselves, but AIUI depending on a javascript one is somewhat painful outside npm world. So I'd personally probably prefer using wasm (the downsides of wasm and Go don't really matter with a local connection).

In any case, TBQH I'm okay with the framerate as it is, so I'm not sure how much time I'll personally invest into improving it in the foreseeable future. This is just a small side-project after all :) (and there are some things that annoy me a bit more).

[bordaigorl]

Yes, the framerate is ok on USB. I was thinking of something simple, for example:

• reading the pen input buffering the events every say 1/10s and calculating the bounding rectangle of the coordinates of the event;
• send in binary format the offset, size, and pixels of that rectangle;
• the js reads the binary data into a byte array
• the js renders the data on a canvas by setting the individual pixels in the interested rectangle
• every now and then a full framebuffer is sent so that approximations of the rectangle are not a problem. Similarly, button presses events could trigger a full framebuffer send.

Then one can play with compression...

[Merovius]

reading the pen input buffering the events every say 1/10s and calculating the bounding rectangle of the coordinates of the event;

This isn't really enough. The frame even occasionally changes without doing anything (consider for example, when you open an e-book - the screen changes from menu to loading screen to pre-render to full render). I'm also not sure if the Linux input API allows multiple readers to open one input device.

As a more general point, I don't really want to take this project that far. I'm not saying it wouldn't be an improvement, that the approximation you'd get isn't enough in practice or that it's not worth doing. But right now I'm intentionally keeping this simple and to the point. If you'd like to take it further, feel free to fork and use it as a starting point for your own ideas, though :) It's open source for a reason :)

[bordaigorl]

Sure, just putting the idea out there ;) This is already incredibly useful as it is, thank you for sharing it!

[Merovius]

I'm going to re-open this - not to say I'm actually going to do it myself, but because there's some useful new information to keep at hand for reference. Recently another project was posted on reddit for the same usecase srvfb serves. Code lives here. What they do is read /dev/fb0 in a loop, stream the data through a compressor and via ssh into an ffmpeg process. Technologically, that's very similar to having a js-decoder (it's going to use less resources because ffmpeg is more efficient, but on the host we have loads of CPU, so that should be fine). So, this actually gives a pretty good estimate for what would be achievable in terms of performance.

One of the reddit-users was so kind to provide a side-by-side comparison: Once with direct streaming and once with proxy-mode. It's pretty clear, I think, that srvfb has significantly higher latency and a lower framerate.

Proxy-mode at least could definitely be sped up significantly, simply by not using the M-JPEG format. On the host, we have enough resources to convert it into a more efficient format that the browser can then play. It would still probably suffer from higher latency, but it might be fine.

As I said, I don't know if I actually will do anything about this. I'd likely be fine piping the data through ffmpeg optionally, in proxy-mode. I still don't know if I want a full js-decoder, even if the benefits are significant. But if nothing else, this provides an alternative for people who are dissatisfied with the performance of srvfb. :)

[torwag]

What do you believe is the bottleneck for the latency?

[Merovius]

I'm not sure, I can only provide educated guesses:

1. A surprisingly large part of it is copying the framebuffer. I'm not sure why, but it takes a while to read it in its entirety. Of course this can't explain differences between implementations.
2. Since we use HTTP and reStream doesn't, it is plausible that introduces additional latency for en- and decoding and headers and such.
3. I think the actually largest chunk is probably due to the MJPEG format. We are seriously abusing it in srvfb, I don't believe it's meant for that. In particular, I still don't understand when exactly chrome updates the image, but it definitely delays by at least a frame - which is a lot, if you only have a couple frames a second.

I don't really have a lot of plausible guesses other than these. I'm pretty sure about 3 - I remember playing around with printing out when we start reading the framebuffer and start writing it out and all of that and I do remember that the image was at least a frame behind. Given how significant that was (and that I had no real intent to change that) I didn't really investigate further.

As I said, performance was never really a super high priority for me.

[bordaigorl]

I have an update: I've been testing reStream and it seems that the main speedup is due to using lz4 to compress the stream efficiently on the remarkable. Since using ffplay for display wasn't really working for me, I reimplemented the approach of reStream in python PyQt5 (including the LZ4 decompression) and it works really well.

https://github.com/bordaigorl/rmview

This way you get good framerate (I get between 5 and 10 fps) and a decent GUI. An extra feature I added is a little red dot that tracks the position of the pen when lifted so you can easily point at things while screencasting (can be disabled through settings).

[pl-semiotics]

@bordaigorl I left this comment on #10, since I saw a mention of damage tracking there, but the approach mentioned there may be of interest to you if you are working on performance enhancements for streaming the reMarkable screen.

If damage information is available, I think VNC/RFB is probably the best ~standard protocol for streaming the framebuffer update information---it sends only the changed pixels, and ZRLE encoding is both quite cheap and effective on the solid background colors that we often see on the tablet. When I want a client other than a normal VNC viewer, I've just been ingesting the data into a gstreamer pipeline (via https://github.com/peter-sa/gst-libvncclient-rfbsrc), which gives a decent range of output/transcoding options.

[stefanoscarponi]

Very helpful. I embeed it on Nativefier and the result is very nice.

Stefano