Alright, here's the big one. This PR introduces a complete rewrite of the partial redraw functionality of terminal mode.
It also implements a new drawing mode for the drivers. It's only in IT8951 for now, but in theory other panels might support it.
TL;DR: the overhead of papertty's logic has been reduced by 90+% in many common scenarios, such as while typing a command, and even more so in other scenarios involving multiple screen elements being modified
Image diffing
The main bottleneck being addressed by this PR is the use of image diffing.
Terminal mode currently performs partial draws by using img_diff to compare two images.
So what happens is that a full-screen image is built, img_diff is used to compare it to the previous frame, and the calculated difference between them is drawn to the panel.
This works, but the performance overhead is quite large.
This becomes especially noticeable when using a high-res panel and a low-spec raspberry pi.
It's still measurable (but not as noticeable) with higher-spec raspberry pis as well.
For the majority of my testing I've been using a waveshare 10.3" HD panel with a resolution of 1872 x 1404.
Performing img_diff on a rpi 4b takes around 200-250ms.
Performing img_diff on a rpi 2b takes around 800-1200ms.
So it is far more noticeable on the 2b, but it's still a fair delay on the 4b.
The big change in this PR is to replace that image diffing with text diffing.
Comparing 50 lines of text is significantly faster than comparing 2.6 million pixels of image data.
The testing and progress of this was somewhat covered here https://github.com/joukos/PaperTTY/issues/107
For the rpi 2b, the change from image diffing to text diffing resulted in a drop of latency from 800->200ms.
For the 4b, the change dropped latency from 200-250ms -> 8-12ms.
*Note that these numbers are only for the papertty logic and I think the SPI writes.
They do not include panel refresh time.
Also note that these numbers are for small differences on the screen, such as typing.
They do not represent large screen changes, such as scrolling.
Multi writes
The second change this PR introduces is currently specific to the IT8951 panel.
It's not directly related to the above changes in papertty.py, but it does require changes to papertty.py in order to take advantage of it, hence I've included it in this PR.
This change is to add support for multiple draws in a single panel refresh.
This cuts down on driver overhead by allowing for multiple small images to be drawn in one go instead of a single big image.
For example, consider a writing application such as wordgrinder.
The text is typed in the middle of the screen.
Separately, there's a word count down the bottom right of the screen.
Each one only receives minor changes, but they are on different parts of the screen.
The old way of handling this would be to draw them both and the space in between.
With text in the middle of the screen and the word count down the bottom of the screen, this would mean effectively drawing half of the screen. (over 1 million pixels on my panel)
With 1bpp mode enabled, that's still 1 million bits, or 122KB, to write over SPI.
Writing the two images separately (the old way) would mean much smaller images.
For example, if you're using a size 30 font, and you type a 5-letter word, then you would have 2 images: a 160x32 image for the text change, and a 32x32 image for the word count.
That's 6,144 bits, or 768 bytes. Obviously a very significant reduction.
The problem then is that each panel refresh has its own overhead.
With A2 mode, the panel refresh is 120ms.
So doing 2 writes would save on the bits written, but add latency by doing 2 refreshes.
With this PR, those two writes would be performed in a single panel refresh.
This is because with IT8951 (and possibly other panels) it's possible to set the coordinates on the screen, write data, change the coordinates, write data, THEN do a refresh.
So you can write multiple images, but still only perform a single refresh.
(This is not specific to IT8951, and it's not specific to the terminal mode changes. However, the terminal mode changes have been written to support this new mode, and I only have an IT8951 panel for testing, so...)
Summary
All up, this should result in significant latency reduction, depending on the use case.
Things like full-screen refreshes or scrolling are still slow, of course.
But common operations like simply typing are significantly faster.
Currently the changes are on by default, but let me know if you think a flag should be added to enable/disable this functionality.
I've tested this on all orientations, with all variations of portrait, flipx and flipy.
I've also tested with multi_draw on and off.
All testing has been done on the same panel, however, due to lacking hardware.
Alright, here's the big one. This PR introduces a complete rewrite of the partial redraw functionality of terminal mode. It also implements a new drawing mode for the drivers. It's only in IT8951 for now, but in theory other panels might support it.
TL;DR: the overhead of papertty's logic has been reduced by 90+% in many common scenarios, such as while typing a command, and even more so in other scenarios involving multiple screen elements being modified
Image diffing
The main bottleneck being addressed by this PR is the use of image diffing. Terminal mode currently performs partial draws by using img_diff to compare two images. So what happens is that a full-screen image is built, img_diff is used to compare it to the previous frame, and the calculated difference between them is drawn to the panel.
This works, but the performance overhead is quite large. This becomes especially noticeable when using a high-res panel and a low-spec raspberry pi. It's still measurable (but not as noticeable) with higher-spec raspberry pis as well.
For the majority of my testing I've been using a waveshare 10.3" HD panel with a resolution of 1872 x 1404. Performing img_diff on a rpi 4b takes around 200-250ms. Performing img_diff on a rpi 2b takes around 800-1200ms. So it is far more noticeable on the 2b, but it's still a fair delay on the 4b.
The big change in this PR is to replace that image diffing with text diffing. Comparing 50 lines of text is significantly faster than comparing 2.6 million pixels of image data.
The testing and progress of this was somewhat covered here https://github.com/joukos/PaperTTY/issues/107 For the rpi 2b, the change from image diffing to text diffing resulted in a drop of latency from 800->200ms. For the 4b, the change dropped latency from 200-250ms -> 8-12ms.
*Note that these numbers are only for the papertty logic and I think the SPI writes. They do not include panel refresh time. Also note that these numbers are for small differences on the screen, such as typing. They do not represent large screen changes, such as scrolling.
Multi writes
The second change this PR introduces is currently specific to the IT8951 panel. It's not directly related to the above changes in papertty.py, but it does require changes to papertty.py in order to take advantage of it, hence I've included it in this PR.
This change is to add support for multiple draws in a single panel refresh. This cuts down on driver overhead by allowing for multiple small images to be drawn in one go instead of a single big image.
For example, consider a writing application such as wordgrinder. The text is typed in the middle of the screen. Separately, there's a word count down the bottom right of the screen. Each one only receives minor changes, but they are on different parts of the screen.
The old way of handling this would be to draw them both and the space in between. With text in the middle of the screen and the word count down the bottom of the screen, this would mean effectively drawing half of the screen. (over 1 million pixels on my panel) With 1bpp mode enabled, that's still 1 million bits, or 122KB, to write over SPI.
Writing the two images separately (the old way) would mean much smaller images. For example, if you're using a size 30 font, and you type a 5-letter word, then you would have 2 images: a 160x32 image for the text change, and a 32x32 image for the word count. That's 6,144 bits, or 768 bytes. Obviously a very significant reduction.
The problem then is that each panel refresh has its own overhead. With A2 mode, the panel refresh is 120ms. So doing 2 writes would save on the bits written, but add latency by doing 2 refreshes.
With this PR, those two writes would be performed in a single panel refresh. This is because with IT8951 (and possibly other panels) it's possible to set the coordinates on the screen, write data, change the coordinates, write data, THEN do a refresh. So you can write multiple images, but still only perform a single refresh.
(This is not specific to IT8951, and it's not specific to the terminal mode changes. However, the terminal mode changes have been written to support this new mode, and I only have an IT8951 panel for testing, so...)
Summary
All up, this should result in significant latency reduction, depending on the use case. Things like full-screen refreshes or scrolling are still slow, of course. But common operations like simply typing are significantly faster.
Currently the changes are on by default, but let me know if you think a flag should be added to enable/disable this functionality.
I've tested this on all orientations, with all variations of portrait, flipx and flipy. I've also tested with multi_draw on and off. All testing has been done on the same panel, however, due to lacking hardware.