PhilDevProg
commented
1 year ago The OSK from GNOME uses the text_input_unstable_v3 Wayland protocol for auto opening. I don't know how the Maliit keyboard used by KDE Plasma works though...
Open flxzt opened 1 year ago
PhilDevProg
commented
1 year ago The OSK from GNOME uses the text_input_unstable_v3 Wayland protocol for auto opening. I don't know how the Maliit keyboard used by KDE Plasma works though...
hmpthcs
commented
1 year ago So happy to see this issue addressed. Rnote was one of the very first applications I tested on my pinenote. Indeed the lag is unacceptably high for use on the device. I have been very impressed with what xournalpp has managed to do to reduce latency. These are the relevant commits / issues, I believe, largely thanks to the work of @hrdl-github:
As I really like rnote as an app, I had hoped to investigate the rnote code to look for similar potential fixes..Still working to understand the architecture. Is there a place I can find more info? Which files in particular would be good candidates to look into? I am also happy to test on my pinenote if it helps.
hrdl-github
commented
9 months ago My patches for xournalpp mostly reduce the extent of damage regions by shrinking repaints to the extent of new strokes and hiding the cursor. At the moment creating a dot (1-3 pixels in extent) with the cursor always being disabled creates around four 5x5 mostly overlapping damage regions, which in principal can be reduced even further.
For rnote I get around 200-400 damage regions of various sizes (no cursor while drawing, but the cursor is updated when entering the drawing area and its changing contact state):
hrdl-github
commented
9 months ago I just noticed that I had a sidebar open. Here's another dataset and its plot when the sidebar's closed:
hrdl-github
commented
9 months ago Experimenting with WAYLAND_DEBUG=1 is probably good enough when working on this. I see a lot of repaints for the green cross that's in the top-left corner by default. By default its fourth quadrant is visible, leading to a 7x7 damage region every time a subregion of the drawing area is repainted, . When moving the drawing area this becomes as large as 14x14 pixels. Upon release of the mouse button I see two wl_surface.damage_buffer for the entire drawing region and the damage to the 64x64 cursor surface when the cursor reappears.
I haven't been able to compile rnote yet as cmake terminates with SIGABRT. I might look into this at some point.
hrdl-github
commented
9 months ago I've had some success removing some repaints (https://github.com/hrdl-github/rnote/tree/redraw , the initial debug build takes around 1.5 hours to build on a PineNote) for testing purposes. Before looking into upstreamable changes I will need a better understanding of how the rnote handles concept like extent of padding (e.g. going from a sequence of input events to an image of limited extent or a render node). Do you have time to provide some input on this, @flxzt ?
flxzt
commented
9 months ago Thanks for looking into this!
I am struggling a bit to understand the logs and the plots. What are the axes on those?
Okay, I'll try to point you to the relevant code. The input events start to get processed here and get dispatched to the current pen (brush) here. The brush implementation then collects events and draws a prediction immediately to the gtk4::Snapshot here (the piet rendercontext for that is created here ).
The brush adds new segments of the current stroke to the store as soon as it can and regenerates the rendering for them here. The function append_rendering_last_images() rerenders only the last segments of the stroke and generates rendernodes for them here these rendernodes are then appended to the gtk4::Snapshot when drawing the widget here.
I can think of three sources for excessive repaints. As you mentioned the indicator cross is always rerendered without any caching, disabling it could be done by commenting out this line. Another source could be the background tiles, they are regenerated here. The last source could be the previously mentioned drawn overlay for the current pen, although that should be limited to the bounds it only really needs.
hrdl-github
commented
9 months ago I am struggling a bit to understand the logs and the plots. What are the axes on those?
Those are pixel coordinates, which would also be reported by wayland at output scale=1.0.
Thanks for the pointers. https://github.com/flxzt/rnote/blob/e4c75e0df6e6ab24babe435c16d15d6bf330a1f2/crates/rnote-engine/src/drawable.rs#L55 I had actually missed. I'll start digging around from there to find out where the padding comes from.
You are right about the indicator cross. I think another source for full drawing area repaints is https://github.com/hrdl-github/rnote/blob/e4c75e0df6e6ab24babe435c16d15d6bf330a1f2/crates/rnote-engine/src/document/mod.rs#L203, which gets called every time a stroke is finished.
hrdl-github
commented
9 months ago Some sources of padding include https://github.com/flxzt/rnote/blob/e4c75e0df6e6ab24babe435c16d15d6bf330a1f2/crates/rnote-engine/src/render.rs#L304 and https://github.com/flxzt/rnote/blob/e4c75e0df6e6ab24babe435c16d15d6bf330a1f2/crates/rnote-engine/src/render.rs#L307-L308. This gets amplified when the zoom factor is > 1. For example, attotal_zoom=5.352124400138854, and 2 or 3 new stroke segments' Aabb { mins: [157.486 118.486], maxs: [158.700, 119.497] } (after loosening), I get a cairo image of size 6x5. The resulting RenderNode { bounds: Rect { x: 157.48618, y: 118.4857, width: 1.2142334, height: 1.0112457 }, node_type: TransformNode } at https://github.com/flxzt/rnote/blob/e4c75e0df6e6ab24babe435c16d15d6bf330a1f2/crates/rnote-engine/src/render.rs#L278 is still smaller than the reported damage region wl_surface@34.damage_buffer(516, 267, 12, 11). It looks like often a value comparable to the zoom factor is added in either dimension.
Lost connection to Wayland compositor.. When patching wayland to use a larger input queue using https://gitlab.freedesktop.org/wayland/wayland/-/issues/159#note_1074515 , rnote no longer crashes, but seems to block for a few seconds instead until finally rendering all strokes from the accumulated input events. Do you have any thoughts on this or suggestions on how to profile this behaviour? I don't have a lot of experience with rust.hrdl-github
commented
9 months ago About 2.: it looks like GTK's gsk_renderer_render blocks and is stuck somewhere in mesa for 5-10 seconds, which will need some more debugging. I don't have an overview over GTK4's / rnote's multithreading model yet, but I feel like input processing typically shouldn't stop.
hmpthcs
commented
6 months ago Not sure which commits are responsible for this, but wanted to provide an update:
I just installed v0.9.3 from the arch repo, and latency is drastically reduced. I should also note that I am using wlroots-git and sway-git packages, as well as a fairly up to date mesa package (v23.3.3).
I compared rnote to xournalpp, and rnote subjectively does remain slightly slower..and there were some crashes when working with .pdfs. Still, while not perfect by any means, I'm very happy to report that rnote is now quite useable on e-ink. UI and touch interactions were very smooth, and drawing is functional. Props to any and all who worked on these issues.
Will try to get some numbers up here for reference soon...for now though I'm eager to see what a build from source is able to do..
flxzt
commented
5 months ago I currently am able to build and test with a PineNote myself but I am currently using the debian image with gnome. Is it dependent on the epd driver waveform settings or can you determine what needs to be updated to improve screen refresh latency?
I also added a script to deploy remotely here to be able to build on another arm machine and deploy it to the pinenote. Works decently well with a raspberry pi, build times are not too brutal.
hmpthcs
commented
4 months ago Bravo, 0.10.1 shows even better performance. At this point I suspect much of the remaining latency may result from libinput and wayland bottlenecks. Really not sure, but am going to play with various compositors and graphics environment variables in the meantime. Also hoping to find time to try working on a patch to give the option to remove the drawing cursor..
The deploy script is a nice addition as well, thank you!
Is your feature request related to a problem? Please describe.
The app in it's current state works very well for 2-in-1 tablets, good for external drawing tablets and decent for touch-only tablets.
There is a new, very exciting class of devices: open Epd tablets with fast-response times, such as the PineNote/ Bigme. I recently had the chance to test the app on a PineNote myself, and while a lot of features work relatively well, I noticed some papercuts where we could improve the experience significantly:
I am sure there are more things that could be improved, so I plan to add more here over time.
Additional Context