Implemented more functions for better compatibility with LVGL

[landonr]

Are these all LVGL related, or is some touch screen? do you have an example config?

[expaso]

Hi @@.***>,

Sorry, I just saw I PRd the wrong way around. I was in the process of pulling in your changes to my fork of your repo.

Basically I adapted your drivers for the display and the touchscreen of the Lilygo T-Display S3 to provide better compatibility with LGVL (I used it in this project VibroTactile Gloves against Parkinson's disease | Elektor Magazinehttps://www.elektormagazine.com/labs/vibrotactile-gloves-against-parkinsons-disease).

The issue is more of an ESPHome issue actually. I basically needed 2 things:

Drawing into own framebuffer, and flip it:

LVGL is a library that is drawing directly in it's own allocated framebuffers. The ESPHome display model is more of a model that gives you methods to draw into a buffer allocated by the display driver. For performance reasons, I needed a way to let LVGL handle it's own framebuffer, and when it's time to show that framebuffer on the screen, hand that buffer over to your displaydriver to copy it to the LCD, while LVGL can do drawing of the next frame on a secondary framebuffer.

This way I could use the multiple cores on the ESP32: The main thread lets LVGL do it's drawing, and a secondary thread (that runs on the other core) that copies over the framebuffer to the display.

To make that work, I added a few settings to disable the internal framebuffer (because LVGL allocates it's own buffers), and to use ASYNC_IO (i.e. use an internal thread on a separate core to do the actual copying of the data so the main core is free for drawing again). Because only LVGL knows which portion of the screen actually needs updating, the updateArea function was added, so the driver does not have to copy over the entire frame each time. This is tremendous performance saver, leading to more then 110 FPS on this device.

Note: You can even mix and match both. So, you could let LVGL do it's stuff, and write with the normal ESPHome drawLine and drawPixel functions on the screen at the same time (don't know why you would need this, but I left it intact).

ESPHome does not have an API or interface for this kind of buffer-flipping of area-management

Touchscreen:

Because my device was battery powered, I added some extra means of telling the touchscreen driver IC to go to sleep, so it consumes less energy. The change to add IRAM_ATTR to the interrupt routine is needed to 'wake' the ESP on touchcreen events.

Basically, the touchscreen chip is capable of going to a lower power state ('sleep') where it polls the display less frequently, but it still can recognize some built-in gestures in this mode, waking the ESP from sleep to handle this.

And now the big but:

I saw there were quite some changes going on around ESPHome display and displaybuffer handling lately. If you like, I could try to adapt all these changes and update this PR.

Have you opted for integrating your code into regular ESPhome? This way, we could perhaps push dis ESPHome display model to be more inline with the scneario's above. I think it would be a huge benefit for multiple display's being able to do ASYNC drawing and/or delegate drawing to a specific library like LVGL.

Let me know your thoughts!

BTW, nice drivers! I enjoyed working on them!

Kind regards,

Hans

---

Van: Landon Rohatensky @.> Verzonden: zaterdag 6 april 2024 04:59 Aan: landonr/lilygo-tdisplays3-esphome @.> CC: Hans van Essen @.>; Author @.> Onderwerp: Re: [landonr/lilygo-tdisplays3-esphome] Implemented more functions for better compatibility with LVGL (PR #61)

Are these all LVGL related, or is some touch screen? do you have an example config?

—

Reply to this email directly, view it on GitHubhttps://github.com/landonr/lilygo-tdisplays3-esphome/pull/61#issuecomment-2040908172, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGIIE7EPSO47JI2MQFB5YCTY35QJNAVCNFSM6AAAAABFNCPTW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBQHEYDQMJXGI.

You are receiving this because you authored the thread.

[landonr]

Regular support for these displays should be in ESPHome, once I get a working config, I'll deprecate this repo #60

I still don't get how you're building LVGL, do you have a config I can try running?

[expaso]

Yeah sure!

I will create a small demo with a config for you this week, without too much clutter, it that allright?

[clowrey]

LVGL will be in mainline ESPhome soon 😁 I've tried it and works great! Just really need some PSRAM.. https://deploy-preview-3510--esphome.netlify.app/components/lvgl.html#:~:text=to%2016.-,buffer_size,-(Optional%2C%20percentage https://gist.github.com/clydebarrow/ef89e9a93bd44771483b9144ae9042a1 https://gist.github.com/clydebarrow/9d2cffd739cb844e9d6d5005fd29518d

[expaso]

Yeah I checked out the lvgl component, and I use it in my project.

It's a nice start, but it performance was not good enough, due to the fact that the communication between the display-driver and the LVGL component was not optimal. This leads to sub-par FPS. Thats why I created the multi-threaded buffer handling, and richer interface between the driver and lvgl to optimize what exactly has to be drawn.

Edit: I am giving waaaay to less credit here, because the LVGL component as it has currently landedin ESPHome is a HUUGE step for UI. It wasn't there when I needed it back in november so I had to do all this stuff by myself.

And there were other factors too, like I created my whole interface in Squareline studio, and I wanted to export all those C-files to a folder, where ESPHome would pick them up, without modification.

This is what I did with it: https://youtu.be/RJwAdNKXpX8?si=0WGuvMw0ej2FJtB9&t=57

But I am more then happy to work with you guys to see if we squeeze the maximum awesomeness out of this combination. Because LVGL with ESPHome and these touch displays works really really great!

Check: https://youtube.com/shorts/McAPOdtXWlw?feature=share

[clowrey]

@Expaso Awesome!! I was just excited to share that we will have something better than the default older "lambda" graphics methods in ESPhome. What you have done looks amazing too!! And you're right the current implementation is not the fastest - it also basically NEEDS PSRAM - not sure if yours somehow can use less ram for buffers - I am basically just a user of these components and ESPhome not a developer myself, any and all improvements and options welcomed by me :)

[landonr]

I'm excited to try this out! This is a huge improvement to esphome!

Can you run clang format to get the lint to pass?

@guillempages do you want to take a quick look here?

[expaso]

@guillempages , Thanks for the review! I really appreciate!

I have a question tough.. @landonr mentioned that this display chip is landing in mainstream ESPHome soon, and this repo would be deprecated.

Is this PR worth it's value then?

Also in my opinion, from an architectural perspective maybe we should seek to implement the ASYNC_IO functionality on a lower level, like the ESPHome's Display class, because a lot of display drivers inheriting from that could benefit from this functionality.

Another reason is that we need class-interface changes to play nicely with LVGL. What LVGL wants is:

a) Allocate it's own buffers. LVGL has very good reasons for this. The buffers could be single, or double, don't need to be the entire framesize perse, and prefer special capabilities. This means that the display driver does NOT need to allocate it's own buffers. Here you see this for double buffers on my ESP32:

       auto *buf1 = (lv_color_t *)heap_caps_malloc(sizeof(lv_color_t) * len, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
       auto *buf2 = (lv_color_t *)heap_caps_malloc(sizeof(lv_color_t) * len, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);

As you see, these are no ordinary mallocs. The MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT capabilities prefer FAST Internal RAM, instead of PSRAM. (they will fallback to PSRAM tough if not enough internal RAM), because of FPS performance. Let the slow PSRam be used for other non-critical stuff.

b) Being able to hand over a buffer, and indicate what exact portion of that buffer needs to be updated to the screen (updateArea). Perhaps only a small portion of the framebuffer was actually updated by LVGL, so only those pixels need to be copied over to the actual screen.

c) A callback to know when the display-driver is done flushing the freshly handed-over framebuffer to the screen, so the next frame can be presented.

Perhaps we should seek to consult JesseRocks and find out what his roadmap for disdplay-drivers is like? Maybe we can work towards more standardization.

What is your view on this?

[guillempages]

@guillempages , Thanks for the review! I really appreciate!

I have a question tough.. @landonr mentioned that this display chip is landing in mainstream ESPHome soon, and this repo would be deprecated.

Is this PR worth it's value then?

Well, I didn't comment it before, but this PR is actually doing two (or actually three) things: It's improving the performance for the display rendering, it's allowing the sleep mode for the display and it's allowing the sleep mode for the touchscreen.

I'd say the touchscreen changes are not worth it, since there is already an official implementation in upstream of the CST chip used. (Unless you are adding some functionality not present in ESPHome, and don't want to add it there ;-) )

If I am not mistaken, the implementation for the display driver that is done upstream is for the OLED version of the t-display-S3; which is pretty different than this one. That said, I haven't really checked in detail; it might be, that this one also already works there.

Also in my opinion, from an architectural perspective maybe we should seek to implement the ASYNC_IO functionality on a lower level, like the ESPHome's Display class, because a lot of display drivers inheriting from that could benefit from this functionality.

This would benefit other displays as well, but thenm, it would need to be done on the ESPHome repo ;-)

Another reason is that we need class-interface changes to play nicely with LVGL. What LVGL wants is:

a) Allocate it's own buffers. LVGL has very good reasons for this. The buffers could be single, or double, don't need to be the entire framesize perse, and prefer special capabilities. This means that the display driver does NOT need to allocate it's own buffers. Here you see this for double buffers on my ESP32:

       auto *buf1 = (lv_color_t *)heap_caps_malloc(sizeof(lv_color_t) * len, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
       auto *buf2 = (lv_color_t *)heap_caps_malloc(sizeof(lv_color_t) * len, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);

As you see, these are no ordinary mallocs. The MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT capabilities prefer FAST Internal RAM, instead of PSRAM. (they will fallback to PSRAM tough if not enough internal RAM), because of FPS performance. Let the slow PSRam be used for other non-critical stuff.

b) Being able to hand over a buffer, and indicate what exact portion of that buffer needs to be updated to the screen (updateArea). Perhaps only a small portion of the framebuffer was actually updated by LVGL, so only those pixels need to be copied over to the actual screen.

c) A callback to know when the display-driver is done flushing the freshly handed-over framebuffer to the screen, so the next frame can be presented.

Perhaps we should seek to consult JesseRocks and find out what his roadmap for disdplay-drivers is like? Maybe we can work towards more standardization.

What is your view on this?

There are currently some efforts done by Clydebarrow to improve the performance of some display drivers, and to add LVGL support. Jesse would welcome any performance improvements on ESPHome, AFAIK, I'm not sure that there is an explicit roadmap (I don't think so). Clyde and NielsNL have some ideas on how to proceed on a display front; you might want to join the discussions on discord: https://discord.com/channels/429907082951524364/1119046373543780363

This repository initially had just a description on how to use TFT_eSPI for this display in combination with ESPHome, but without the whole display implementation; i.e. just allowing C++ drawing. At some point, I reworked the code a bit to be able to use it as a standard ESPHome display, but did not focus (much) on optimization; just on having it work without much hassle. The repo as-is (i.e. using TFT_eSPI) will most probably never get merged into ESPHome; since the project frowns upon adding (big) external libraries unless really needed. If you have ideas on improving the performance here (as in this PR), you are welcome to submit them, and we will review them, as long as they do not break standard ESPHome functionality.

[landonr]

@clydebarrow can you chime in? Are you working on mainlien support for TDisplay S3 Amoled? It'd like to make devices pages for TDisplay S3 (tft and amoled)

[clydebarrow]

@clydebarrow can you chime in? Are you working on mainlien support for TDisplay S3 Amoled? It'd like to make devices pages for TDisplay S3 (tft and amoled)

T-Display S3 AMOLED (quad-SPI) is already supported in ESPHome along with the T4-S3. TFT T-Display S3 (which I think is an i8080 interface) is not yet supported, but it's a WIP.

[h3ndrik]

This doesn't seem to compile (anymore):

[...]
Compiling .pioenvs/boombox/src/esphome/components/tdisplays3/t_display_s3.cpp.o
In file included from src/esphome/components/tdisplays3/t_display_s3.cpp:1:
src/esphome/components/tdisplays3/t_display_s3.h:31:8: error: 'void esphome::tdisplays3::TDisplayS3::updateArea(int16_t, int16_t, int16_t, int16_t, uint16_t*, void (*)())' marked 'override', but does not override
   void updateArea(int16_t x1, int16_t y1, int16_t x2, int16_t y2, uint16_t *buffer, void (*ready_callback)(void)) override;
        ^~~~~~~~~~
*** [.pioenvs/boombox/src/esphome/components/tdisplays3/t_display_s3.cpp.o] Error 1
========================= [FAILED] Took 164.49 seconds =========================

(And unfortunately, seems support for the displays with a parallel bus hasn't made it into ESPHome. I don't see any current work on that.)

[clydebarrow]

support for the displays with a parallel bus hasn't made it into ESPHome. I don't see any current work on that.)

It's available as an external component. I closed the PR for reasons I won't discuss here.

[expaso]

Yeah, I saw that display-code was really in flux last versions of ESPHome, breaking this PR, and was not going into the direction I need to keep the performance optimal when used with LVGL.

@clydebarrow You can indeed abandon this PR. I have pulled everything I need to external components myself until ESPHome has a more mature display driver architecture.

[h3ndrik]

Hmm. Bummer. https://github.com/esphome/esphome/pull/6537 seems to require ESP_IDF, and I need the Arduino platform for a media_player, which has already been discussed in some of the issues/PRs. And I mean it's bound to be out of date at some point, too...

@clydebarrow You mentioned seperating out the PR in the esphome repo, put in some effort and then closed it on Aug 7. I suppose you're not working on that anymore? Are there things to know about when someone else attempts to get that running or incorporated into esphome? Any technical barriers / personal matters? Or were you just unable to put in the effort to clean it up and get it accepted in esphome after all of the lvgl stuff? I mean you already said you won't discuss it here. So feel free not to answer. I'd just like not to waste too much time myself and run into the same obstacles.