Add display driver for RGB dot clock displays on ESP32-S3 port

[kmatch98]

The ESP32-S3 has a built-in LCD controller, including the ability to drive parallel (8080) and dot-clock driven displays. If CircuitPython can directly drive RGB displays, it may allow the use of lower-cost displays. I'm interested to understand whether using CircuitPython to drive RGB displays is feasible or if there are issues that make it not worthwhile.

I verified the Espressif RGB LCD demo is working on the N8R8 version of the Espressif development board.

I'm interested to explore whether this is a useful addition to CircuitPython but would appreciate guidance on what is required to connect it into CircuitPython.

Here's the first of many questions:

1. The IDF has some high level constructs for connecting the display. Should these be used, or will I need to burrow down closer to the hardware to make the links to CircuitPython.
2. Currently, the displayio makes a framebuffer, and the "bus" is just a pipe for shoving data down. In contrast, with the RGB LCD driver, it defines its own display buffer. Do I need a separate displayio framebuffer and a separate LCD framebuffer, or can these be the same? However, displayio also has the option to turn refresh OFF. To keep that functionality, that will require separate displayio and LCD frame buffers. Perhaps the "bus" for RGB-LCD is a function that copies the displayio framebuffer into the LCD framebuffer with any conversion required for RGB565.
3. The framebuffer will likely reside in PSRAM. I see some mention that the LCD peripheral can use DMA to access the PSRAM without requiring clock cycles. However, if the LCD peripheral is always heavy DMA-ing the PSRAM, will that excessively slow down the processor cycles that need to access PSRAM?

All suggestions are welcome, including direction whether this is useful or riddled with problems.

[tannewt]

Here's the first of many questions:

1. The IDF has some high level constructs for connecting the display. Should these be used, or will I need to burrow down closer to the hardware to make the links to CircuitPython.

I think it depends on how you plug into displayio.

2. Currently, the displayio makes a framebuffer, and the "bus" is just a pipe for shoving data down. In contrast, with the RGB LCD driver, it defines its own display buffer. Do I need a separate displayio framebuffer and a separate LCD framebuffer, or can these be the same? However, displayio also has the option to turn refresh OFF. To keep that functionality, that will require separate displayio and LCD frame buffers. Perhaps the "bus" for RGB-LCD is a function that copies the displayio framebuffer into the LCD framebuffer with any conversion required for RGB565.

I would only have one framebuffer so that you can support larger displays. displayio also has the display state in it's object tree so it doesn't require it's own framebuffer. Does the IDF just provide ways of drawing to it's framebuffer? That'd be similar to how FourWire writes areas of the framebuffer on the separate display controller chip.

3. The framebuffer will likely reside in PSRAM. I see some mention that the LCD peripheral can use DMA to access the PSRAM without requiring clock cycles. However, if the LCD peripheral is always heavy DMA-ing the PSRAM, will that excessively slow down the processor cycles that need to access PSRAM?

It depends heavily of how frequently the LCD peripheral and the CPU need to access RAM. I wouldn't worry about it now. The functionality will be helpful even if it does slow things a bit.

[kmatch98]

Regarding item 2:

The latest ESP32 idf has a function to copy a bitmap to a location in the RGB LCD frame buffer.

Here is a snippet from the LCD example:

    // copy a buffer's content to a specific area of the display
    esp_lcd_panel_draw_bitmap(panel_handle, offsetx1, offsety1, offsetx2 + 1, offsety2 + 1, color_map);

For displayio, I believe the palette uses 24 bit color, so we will need to do conversion to 16-bit RGB566 before performing this operation. I see there is a built in colorspace for displayio. I’ll need to learn how to use this.

[kmatch98]

I’ve been poking around with the example code and removing the LVGL parts so that I can directly manipulate the calls to draw_bitmap.

Unfortunately based on my testing and some code review, I think that the function esp_lcd_panel_draw_bitmap only provides for transfer of one bitmap for each frame. I don’t think this is adequate for CircuitPython since it draws several rectangles of data in the _refresh_area function.

Here’s the code in the ESP-IDF for the draw_bitmap also copied below.

My understanding of the draw_bitmap code:

1. Checks and adjusts x- and y-ranges
2. Grabs the flag (semaphore) for writing to the display buffer.
3. Copies the data to the display buffer
4. Clears the cache (?)
5. Increments the frame counter.
6. Calls for an LCD refresh (unsure about this last line)

I think I need to divide this into three chunks: “start, copy, end”. That will allow me to do several copy steps before displaying the next frame on the lcd.

static esp_err_t rgb_panel_draw_bitmap(esp_lcd_panel_t *panel, int x_start, int y_start, int x_end, int y_end, const void *color_data)
{
    esp_rgb_panel_t *rgb_panel = __containerof(panel, esp_rgb_panel_t, base);
    assert((x_start < x_end) && (y_start < y_end) && "start position must be smaller than end position");
    // adjust the flush window by adding extra gap
    x_start += rgb_panel->x_gap;
    y_start += rgb_panel->y_gap;
    x_end += rgb_panel->x_gap;
    y_end += rgb_panel->y_gap;
    // round the boundary
    x_start = MIN(x_start, rgb_panel->timings.h_res);
    x_end = MIN(x_end, rgb_panel->timings.h_res);
    y_start = MIN(y_start, rgb_panel->timings.v_res);
    y_end = MIN(y_end, rgb_panel->timings.v_res);
    xSemaphoreTake(rgb_panel->done_sem, portMAX_DELAY); // wait for last transaction done
    // convert the frame buffer to 3D array
    int bytes_per_pixel = rgb_panel->data_width / 8;
    int pixels_per_line = rgb_panel->timings.h_res;
    const uint8_t *from = (const uint8_t *)color_data;
    uint8_t (*to)[pixels_per_line][bytes_per_pixel] = (uint8_t (*)[pixels_per_line][bytes_per_pixel])rgb_panel->fb;
    // manipulate the frame buffer
    for (int j = y_start; j < y_end; j++) {
        for (int i = x_start; i < x_end; i++) {
            for (int k = 0; k < bytes_per_pixel; k++) {
                to[j][i][k] = *from++;
            }
        }
    }
    if (rgb_panel->flags.fb_in_psram) {
        // CPU writes data to PSRAM through DCache, data in PSRAM might not get updated, so write back
        Cache_WriteBack_Addr((uint32_t)&to[y_start][0][0], (y_end - y_start) * rgb_panel->timings.h_res * bytes_per_pixel);
    }
    // we don't care the exact frame ID, as long as it's different from the previous one
    rgb_panel->new_frame_id++;
    if (!rgb_panel->flags.stream_mode) {
        // in one-off mode, the "new frame" flag is controlled by this API
        rgb_panel->cur_frame_id = rgb_panel->new_frame_id;
        rgb_panel->flags.new_frame = 1;
        // reset FIFO of DMA and LCD, incase there remains old frame data
        gdma_reset(rgb_panel->dma_chan);
        lcd_ll_stop(rgb_panel->hal.dev);
        lcd_ll_fifo_reset(rgb_panel->hal.dev);
        gdma_start(rgb_panel->dma_chan, (intptr_t)rgb_panel->dma_nodes);
    }
    // start LCD engine
    lcd_ll_start(rgb_panel->hal.dev);
    return ESP_OK;
}

[kmatch98]

https://user-images.githubusercontent.com/33587466/158075273-1390c44d-9db9-4ff0-889d-67bf2e040df3.MOV

I broke it into three chunks and it’s doing better but not exactly what I wanted.

I have a loop doing 100 “copy” to the buffer before calling “finish”. It appears to operate smoothly during the “copy” phase but stutters at the “finish” phase. Periodically the code will crash due to CPU0 watchdog timeout and cause the processor to restart.

I currently have serial debug statements that is slowing the drawing. If I remove those serial ESP_LOGI debug statements I get a panic crash very early.

I’ll look deeper into my code but will probably need feedback on things to watch out for. I still don’t exactly understand the function of lcd_ll_start and why this is required.

[kmatch98]

https://user-images.githubusercontent.com/33587466/158093649-7e73fd0a-fec6-4854-924a-b45b0bb5d2b0.MOV

It turns out you just need to call lcd_ll_start once and then you can keep writing away to the display buffer to your heart’s content. I added vTaskDelay(pdMS_TO_TICKS(10)); to prevent the crashing.

(So far I’ve hacked into the ESP-IDF and changed some of the structure to accommodate these drawing functions. I’ll eventually need some guidance how to make it into a standalone package.)

[kmatch98]

Here’s a brief summary of what I did to make the bitmap redrawing work.

1. Call lcd_ll_start once in an initialization step.
2. I hacked the ESP-IDF definition of esp_lcd_panel_t to include a new function to do the bitmap copy only. This function checks the x,y bounding box ranges, takes the Semaphore, copies the bitmap into the framebuffer, flushes the cache, then gives back the Semaphore.

Note, the esp_lcd_panel_t Is the generic base class structure for a panel, and each display type sets up the functions required to operate the specific display type. If we need to add a function for the RGB LCD driver, what’s the best way to do this while keeping compatibility with the IDF? To prevent dealing with the peculiarities of each display type in displayio, weshould strive to use only the esp_lcd generic commands in displayio to handle all these different displays. I suspect it is preferable to override the ESP-IDF draw_bitmap and init functions for the RGB LCD. Let me see if I can figure out how to do this. That way we can have only the minimum required differences from the ESP-IDF.

Also, I will clean up my code and post an issue over on the esp-idf GitHub to request clarification, in case I’m missing something.

Main updates to RGB lcd code for ESP32-S3:

• move `lcd_ll_start to the init function.
• remove `lcd_ll_start from draw_bitmap function

Open questions: The code has two modes with or without streaming. I don’t exactly understand how this is used and how this code can be used to increment the frame. I can understand the possible need to draw several things and then later call for the display frame to be updated. But it’s not clear to me whether this code actually achieves that objective and if so, then I need to understand better how to use it. I can’t exactly recall if displayio is smart enough to only draw the pixels it needs to (for example with overlapping groups). If it’s not smart enough to to only draw the top group pixels, then we’ll,probably need to figure out how to update the framebuffer, and then change the frame. I hope there is a way of doing this without having to double buffer.

[tannewt]

displayio should only update pixels that have changed. However, it may do it multiple times if objects provide dirty areas that overlap. The computation result should be the same though.

[kmatch98]

So far I’ve been using ESP-IDF version that's a few weeks old on the Main branch. There was a commit in the last two weeks to address some of the issues I observe above, and mentions some corrections for streaming mode.

I’ll need to pull these lates changes and assess if they clear the path for fast drawing in the base ESP-IDF, as demoed above.

[kmatch98]

I evaluated the recent commits to assist with rgb LCDs and they resolve the issue that I observed above. I'm getting decent speed, here's an example of a 10x10 pixel bouncing box:

https://user-images.githubusercontent.com/33587466/159185726-9074ed31-64d8-4fc6-ad23-4fe3b42abbf6.MOV

I performed an alternate test with 400x400 squares (about half the display size) and am getting around 25 frames per second.

Here's my example code. Adjust box_size to see the impact of redraw size on the performance:

/*
 * SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
 *
 * SPDX-License-Identifier: CC0-1.0
 */

#include <stdio.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_timer.h"
#include "esp_lcd_panel_ops.h"
#include "esp_lcd_panel_rgb.h"
#include "driver/gpio.h"
#include "esp_err.h"
#include "esp_log.h"
#include "lvgl.h"
#include <time.h>

static const char *TAG = "example";

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
//////////////////// Please update the following configuration according to your LCD spec //////////////////////////////
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
#define EXAMPLE_LCD_PIXEL_CLOCK_HZ     (10 * 1000 * 1000)
#define EXAMPLE_LCD_BK_LIGHT_ON_LEVEL  1
#define EXAMPLE_LCD_BK_LIGHT_OFF_LEVEL !EXAMPLE_LCD_BK_LIGHT_ON_LEVEL
#define EXAMPLE_PIN_NUM_BK_LIGHT       39
#define EXAMPLE_PIN_NUM_HSYNC          47
#define EXAMPLE_PIN_NUM_VSYNC          48
#define EXAMPLE_PIN_NUM_DE             45
#define EXAMPLE_PIN_NUM_PCLK           21
#define EXAMPLE_PIN_NUM_DATA0          3  // B0
#define EXAMPLE_PIN_NUM_DATA1          4  // B1
#define EXAMPLE_PIN_NUM_DATA2          5  // B2
#define EXAMPLE_PIN_NUM_DATA3          6  // B3
#define EXAMPLE_PIN_NUM_DATA4          7  // B4
#define EXAMPLE_PIN_NUM_DATA5          8  // G0
#define EXAMPLE_PIN_NUM_DATA6          9  // G1
#define EXAMPLE_PIN_NUM_DATA7          10 // G2
#define EXAMPLE_PIN_NUM_DATA8          11 // G3
#define EXAMPLE_PIN_NUM_DATA9          12 // G4
#define EXAMPLE_PIN_NUM_DATA10         13 // G5
#define EXAMPLE_PIN_NUM_DATA11         14 // R0
#define EXAMPLE_PIN_NUM_DATA12         15 // R1
#define EXAMPLE_PIN_NUM_DATA13         16 // R2
#define EXAMPLE_PIN_NUM_DATA14         17 // R3
#define EXAMPLE_PIN_NUM_DATA15         18 // R4
#define EXAMPLE_PIN_NUM_DISP_EN        -1

// The pixel number in horizontal and vertical
#define EXAMPLE_LCD_H_RES              800
#define EXAMPLE_LCD_V_RES              480

#define EXAMPLE_LVGL_TICK_PERIOD_MS    2

extern void example_lvgl_demo_ui(lv_obj_t *scr);

int color = 0x000000;

static lv_color_t *my_color_map1 = NULL;

int x=0;
int y=0;
int box_size = 10;
int x_velocity=1;
int y_velocity=1;

void app_main(void)
{
    static lv_disp_draw_buf_t disp_buf; // contains internal graphic buffer(s) called draw buffer(s)
    static lv_disp_drv_t disp_drv;      // contains callback functions

    ESP_LOGI(TAG, "Turn off LCD backlight");
    gpio_config_t bk_gpio_config = {
        .mode = GPIO_MODE_OUTPUT,
        .pin_bit_mask = 1ULL << EXAMPLE_PIN_NUM_BK_LIGHT
    };
    ESP_ERROR_CHECK(gpio_config(&bk_gpio_config));

    ESP_LOGI(TAG, "Install RGB panel driver");
    esp_lcd_panel_handle_t panel_handle = NULL;
    esp_lcd_rgb_panel_config_t panel_config = {
        .data_width = 16, // RGB565 in parallel mode, thus 16bit in width
        .psram_trans_align = 64,
        .clk_src = LCD_CLK_SRC_PLL160M,
        // .clk_src = LCD_CLK_SRC_PLL240M,
        .disp_gpio_num = EXAMPLE_PIN_NUM_DISP_EN,
        .pclk_gpio_num = EXAMPLE_PIN_NUM_PCLK,
        .vsync_gpio_num = EXAMPLE_PIN_NUM_VSYNC,
        .hsync_gpio_num = EXAMPLE_PIN_NUM_HSYNC,
        .de_gpio_num = EXAMPLE_PIN_NUM_DE,
        .data_gpio_nums = {
            EXAMPLE_PIN_NUM_DATA0,
            EXAMPLE_PIN_NUM_DATA1,
            EXAMPLE_PIN_NUM_DATA2,
            EXAMPLE_PIN_NUM_DATA3,
            EXAMPLE_PIN_NUM_DATA4,
            EXAMPLE_PIN_NUM_DATA5,
            EXAMPLE_PIN_NUM_DATA6,
            EXAMPLE_PIN_NUM_DATA7,
            EXAMPLE_PIN_NUM_DATA8,
            EXAMPLE_PIN_NUM_DATA9,
            EXAMPLE_PIN_NUM_DATA10,
            EXAMPLE_PIN_NUM_DATA11,
            EXAMPLE_PIN_NUM_DATA12,
            EXAMPLE_PIN_NUM_DATA13,
            EXAMPLE_PIN_NUM_DATA14,
            EXAMPLE_PIN_NUM_DATA15,
        },
        .timings = {
            .pclk_hz = EXAMPLE_LCD_PIXEL_CLOCK_HZ,
            .h_res = EXAMPLE_LCD_H_RES,
            .v_res = EXAMPLE_LCD_V_RES,
            // The following parameters should refer to LCD spec
            .hsync_back_porch = 100,
            .hsync_front_porch = 40,
            .hsync_pulse_width = 5,
            .vsync_back_porch = 25,
            .vsync_front_porch = 10,
            .vsync_pulse_width = 1,
            // .flags.pclk_active_neg = 1, // RGB data is clocked out on falling edge
        },
        // .flags.relax_on_idle = 1, // stream
        .flags.fb_in_psram = 1, // allocate frame buffer in PSRAM
        // .on_frame_trans_done = draw_box,
        .on_frame_trans_done = NULL,
        // .on_frame_trans_done = example_notify_lvgl_flush_ready,
        .user_ctx = &disp_drv,
    };
    ESP_ERROR_CHECK(esp_lcd_new_rgb_panel(&panel_config, &panel_handle));

    ESP_ERROR_CHECK(esp_lcd_panel_reset(panel_handle));
    ESP_ERROR_CHECK(esp_lcd_panel_init(panel_handle));

    ESP_LOGI(TAG, "Turn on LCD backlight");
    gpio_set_level(EXAMPLE_PIN_NUM_BK_LIGHT, EXAMPLE_LCD_BK_LIGHT_ON_LEVEL);

    ESP_LOGI(TAG, "Initialize LVGL library");
    lv_init();

    my_color_map1 = (lv_color_t *) heap_caps_malloc(box_size * box_size  * sizeof(lv_color_t), MALLOC_CAP_SPIRAM);
    assert(my_color_map1);

    for (int i=0; i<(box_size+1) * (box_size+1); i++) {
        my_color_map1[i] = lv_color_hsv_to_rgb(color, 80, 80);
    }

    x=box_size/2+5;
    y=box_size/2+5;

    int size = (box_size+1) * (box_size+1) * sizeof(lv_color_t);

    while (1) {

        for (int i=0; i<50; i++) {
            esp_lcd_panel_draw_bitmap(panel_handle, x-box_size/2, y-box_size/2, x-box_size/2+box_size+1, y-box_size/2+box_size+1, my_color_map1);
            // ESP_LOGI(TAG, "finished copy");
            if ( (x < box_size/2+5) || (x > EXAMPLE_LCD_H_RES- box_size/2-5) ) {
                x_velocity *= -1;
            }
            if ( (y < box_size/2+5) || (y > EXAMPLE_LCD_V_RES- box_size/2-5) ) {
                y_velocity *= -1;
            }

            x += x_velocity;
            y += y_velocity;

            vTaskDelay(pdMS_TO_TICKS(2)); 
            // ESP_LOGI(TAG, "x: %d, y: %d, color: %d, i: %d", x, y, color,i );
        }

        if (color > 330) {
            color = 0;
        }
        else {
            color +=20;
        }

        for (int i=0; i<(box_size+1)*(box_size+1); i++) {
            my_color_map1[i] = lv_color_hsv_to_rgb(color, 80, 80);
        }

    }

}

[kmatch98]

I'm looking at how to adapt the existing code to work with the ESP32-S3 lcd driver code, and I could use some pointers. It is a slog for me to understand some of the code, so I'd appreciate even a few sentences to guide me in the right direction.

The ESP-IDF functions don't really act like a "bus" like the I2C or SPI buses, so I'm trying to figure out an easy way of doing this.

Background:

• The ESP-IDF lcd peripheral library has a _draw_bitmap function that takes a draw window. This takes a pixel buffer and copies it into the display framebuffer pixel in a rectangle specified by (x_start, y_start) and (x_end, y_end):

  rgb_panel_draw_bitmap(esp_lcd_panel_t *panel, int x_start, int y_start, int x_end, int y_end, const void *color_data);

• CircuitPython's displayio uses two main functions to set the drawing area and then send the pixels

  • displayio_display_core_set_region_to_update - This calculates the boundary area for the current rectangle to be drawn. In the case of SH1107 addressing, it sends the start and stop columns and the start row, but not the ending row.
  • _send_pixels - This has inputs of # of pixels and the pixel buffer data to send.

Here's options to transfer this to use _draw_bitmap style:

• Use the SH1107 style addressing code, and catch and decode the start_column, stop_column and start_row from the displayio_display_core_set_region_to_update. Then, whenever the next data is sent, it will have to calculate how many rows are being sent so it can calculate the full window that will be copied. Note: This assumes that _send_pixels always sends a full rectangle of data.
• Create a new _send_window function (use instead of _send_pixels) that sends the copy window boundaries and the data at the same time. Maybe trigger this by a Boolean called "window_addressing".

Questions:

• At first glance, I can't follow all the code in _refresh_area and how it uses the buffer. Does _send_pixels always send a complete rectangle?
• Colordepth: The ESP32-S3 RGB LCD uses 16-bit colors (RGB565). If I set the colorspace.depth to 16, will that take care of everything I need when I go to send pixels to the display? Will I need to do any color conversions to get the data right or is it handled already?

Thanks again for your direction on this!

[kmatch98]

After reviewing the code more, I've decided to create a window_addressing flag, perhaps other controllers will come along that can also use this.

[tannewt]

Can you get access to the framebuffer yourself? If you can then you could use FramebufferDisplay to do the work for you.

• At first glance, I can't follow all the code in _refresh_area and how it uses the buffer. Does _send_pixels always send a complete rectangle?

I believe so. It chops dirty rectangles into subrectangles if needed and then sends the bounds of the subrectangle and then all of the pixel data for the subrectangle.

• Colordepth: The ESP32-S3 RGB LCD uses 16-bit colors (RGB565). If I set the colorspace.depth to 16, will that take care of everything I need when I go to send pixels to the display? Will I need to do any color conversions to get the data right or is it handled already?

Yes, I believe it should give you the format you need. 565 is super common.

[kmatch98]

I’m capturing some discussion from the Discord chat.

FrameBufferDisplay allows for direct writing to a memory buffer. It has similar function as displayio to assess the group structure and draw, but rather than “send”ing data to a bus, it writes directly to a memory space.

The SharpMemoryDisplay and RGBMatrix are examples.

Note: The ESP RGB LCD flushes the cache after writing to the LCD frame buffer, this cache flush can probably be done in the frame buffer swapbuffers function.

Next steps:

• [x] 1. Get CircuitPython to build with a blank DotClockDisplay module directory and files.
• [x] 2. Find how to expose the ESP RGB LCD’s framebuffer memory address up to the FrameBufferDisplay code.
• [x] 3. Hard code where expedient and at least get one pixel to draw.
• [x] 4. Fill in “the rest of the owl”

[kmatch98]

I've got the basic code compiling now and am using make DEBUG=1 and printing to UART to debug the ESP code. I can print to the ESP UART console log using ESP_LOGI(TAG, "esp_lcd_new_rgb_panel 7.2");.

I am getting a crash when the esp_lcd_new_rgb_panel goes to allocate memory for the display framebuffer, right after this line:

rgb_panel->fb = heap_caps_aligned_calloc(psram_trans_align, 1, fb_size, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);

Error message:

Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.

Supposedly the EXCVADDR: 0xe019074d LBEG tells what address was attempted to be loaded and failed.

Based on this issue mentioned in micro python, perhaps the esp-idf code doesn't have any memory left over to allocate: https://github.com/micropython/micropython/issues/5942#issuecomment-616135299.

The current ESP-IDF code doesn't have a mechanism to allow to pass in an existing framebuffer.

If I am understanding the root cause correctly, I see two options:

1. CircuitPython should leave some SPI memory space for the display framebuffer, that the esp-idf code can access that available memory.

2. Make a custom ESP-IDF esp_lcd_new_rgb_panel function that takes a framebuffer as an input. Then CircuitPython should allocate the memory and then pass it to this function.

Suggestions welcome.

[kmatch98]

I found in the espressif port where the heap pointer is allocated.

https://github.com/adafruit/circuitpython/blob/c6bfe54dc8c52574bec7dae4ec1e8113f1e427c7/ports/espressif/supervisor/port.c#L197

It appears that the heap pointer starts at the beginning of the SPI PSRAM. I don’t see any indications of a mechanism to leave some memory space for other functions to allocate memory from. I’m unsure how this can be modified to leave some room for a framebuffer. I need to keep digging.

[kmatch98]

CircuitPython has an espidf module for measuring memory parameters for this build.

If I run the latest version for the Devkit-N8R8 (8MB flash, 8 MB of PSRAM), I get these values. Based on the espidf measurements, the PSRAM is not accessible, but gc.mem_free shows 8MB available. This is unclear to me why:

Adafruit CircuitPython 7.2.3 on 2022-03-16; ESP32-S3-DevKitC-1-N8R8 with ESP32S3
>>> 
>>> import espidf
>>> print(espidf.heap_caps_get_total_size())
330324
>>> print(espidf.heap_caps_get_free_size())
296368
>>> print(espidf.heap_caps_get_largest_free_block())
225280
>>> import gc
>>> print(gc.mem_free())
8194576

For my new build, I turned on the flag in the sdkconfig file CONFIG_SPIRAM_USE_MALLOC=y. This flag allows me to access the SPIRAM in code. This does give access to the SPI PSRAM, as shown by 8.7MB available from heap_caps_get_total_size. But it looks like only 282kB is available to the ESP-IDF.

Specifically, the code in esp_lcd_new_rgb_panel requires access to PSRAM using one of these two sdkconfig flags CONFIG_SPIRAM_USE_MALLOC or CONFIG_SPIRAM_USE_CAPS_ALLOC:

When I use the flag CONFIG_SPIRAM_USE_MALLOC=y, this causes a crash when running the espidf.heap_caps_get_largest_free_block. I get the same LoadProhibited error as when I try and allocate the RBG LCD's framebuffer. I need to better understand the memory management of the PSRAM.

Error message with CONFIG_SPIRAM_USE_MALLOC=y in sdkconfig:

Adafruit CircuitPython 5.3.0-1808-gfcde108d0-dirty on 2022-03-28; ESP32-S3-DevKitC-1-N8R8 with ESP32S3
>>> 
>>> import espidf
>>> print(espidf.heap_caps_get_total_size())
8740362
>>> print(espidf.heap_caps_get_free_size())
282423
>>> print(espidf.heap_caps_get_largest_free_block())

CRASH

Message from the ESP monitor via UART:

Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.

Core  1 register dump:
PC      : 0x4038673f  PS      : 0x00060130  A0      : 0x80387713  A1      : 0x3fcf5130  
A2      : 0xe019105d  A3      : 0x40387750  A4      : 0x3fcf5170  A5      : 0x3fcf518c  
A6      : 0xfffffffc  A7      : 0x00000001  A8      : 0x0000095c  A9      : 0x00000000  
A10     : 0x00000000  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x3fcf5170  
A14     : 0x3fcf5298  A15     : 0x3fcf5300  SAR     : 0x00000011  EXCCAUSE: 0x0000001c  
EXCVADDR: 0xe0191061  LBEG    : 0x400570e8  LEND    : 0x400570f3  LCOUNT  : 0x00000000  

[kmatch98]

I hacked the ESP-IDF to use m_malloc and now I can draw pixels to the screen (hard coded in the init function). I will now try to get draw function working in CircuitPython.

Note: One observation after drawing some pixels to the screen, is that everytime I enter a command in the REPL, the display shifts. I suspect something is going weird with the timing of the RBG signals but I haven’t looked on the scope. Add this to the to-do list.

[kmatch98]

Got to Blinka!

The REPL and Blinka logo are working. The image is offset due to some timing weirdness that shifts the screen whenever CircuitPython does some processing, or if you type in the REPL (see previous update).

Todo: When the code finishes, the display goes weird, doesn't update anymore and doesn't show the REPL.

[kmatch98]

The display shifting bug I observed above has a similar behavior as that described here: https://github.com/espressif/esp-idf/issues/8381

Keep an eye out for when the updates get back ported into the 4.4 version, see this issue to track it: https://github.com/espressif/esp-idf/issues/8620

[kmatch98]

Now that there is some life, I want to keep track of all the issues that I’ve observed so far and any links to relevant information.

• [x] The display glitches whenever CircuitPython is doing some processing.

I think whenever CircuitPython is accessing PSRAM, that reduces the bandwidth available for the LCD peripheral DMA access for redrawing the screen. Here is one related discussion on the ESP32 forum. . In that forum, here is a post that is working to give more control on memory access.

Currently, the LCD peripheral is set to update continuously “stream” mode. There appear to be options to trigger data to be sent to the display only on command. I think that the register lcd_next_frame_en may be an option to control the continuous refresh. . Edit: Nope, didn’t help any.

Perhaps the call to update the screen could be triggered by the FrameBuffer _swap_buffer command. There may need to be some coordination with CircuitPython to indicate when the refresh is complete. (Note: But DMA is supposed to make it where we don’t need coordination.)

• [ ] The display stops updating upon completion of the code.py file execution. Is this how FrameBuffer works, or is something wrong with my module code?

• [ ] Add appropriate pin checking before initializing the display.

• [ ] the de-init function doesn’t seem to be working properly with release_displays.

• [x] Need to incorporate some of ESP-IDF recent commits to allow this to work.

• [x] Need ESP-IDF to be modified to allow passing in an existing framebuffer (allocated from CircuitPython).

• [x] Decide when to make a commit and include my hacks to the ESP-IDF so I don’t lose anything.

[kmatch98]

Thanks to some code from "ESP-sprite" on the ESP32.com forum, along with a few special tweaks, this eliminates the glitches observed above. The "bounce-buffer" code by ESP-sprite actually requires the CPU to write the data to the LCD peripheral through an intermediate buffer. In essence, this locks the processor so that other code does not access PSRAM while the LCD peripheral is accessing PSRAM.

I modified the code so that I can put the display in "non-stream" mode and then call lcd_rgb_panel_start_transmission from CircuitPython in the FrameBuffer swap_buffers function to trigger a refresh. This successfully eliminates any tearing during the LCD display refresh.

Here's a video with a large box (300x300 pixels) and a high velocity between redraws, to show that "tearing" is absent.

https://user-images.githubusercontent.com/33587466/163729604-0fff4431-70ca-4595-aab8-a8aab780d200.MOV

Next steps:

• [ ] Provide feedback to ESP-sprite with my minor updates to the bounce_buffer code.

• [ ] Start work on the list above.

[Pepito-Payet]

@kmatch98

Hello,

I have recycled 2 Crestron TSW 750 and I would like to use them but I can't because of the proprietary system.... I saw that you managed to get some of them to work. Could you please tell me how to do it? It would be a shame to throw them away.... I would like to display sensor data or a web page with my home automation like Jeedom or other. I saw that the TSW 750 has a micro sd card but I don't know if I can install a system inside.

[kmatch98]

Awesome! Glad to see that you’re trying to use these displays. In my work I discarded the original circuit board and just used the display touch panel and added different electronics.

Here is some background on what I’ve done.

https://hackaday.io/project/184118-crestron-tss-752-teardown-rebuild

https://hackaday.io/project/183895-crestron-tsw-732-teardown

If you want to just reuse the touch panel display, first you will need to figure out the pin outs for the display connector. I guess the pin outs are probably the same as the one in my two units but take a close picture of the connector and you can make an educated guess if it’s the same. Also if you find the part number of the display panel you can look for that part number on the “panelook” website.

Next you will need to wire your display to a controller that can drive the RGB signals. I’m using the ESP32-S3 in this project. The ESP-IDF has a sample program that you can try out. But there will be a lot of wires to connect! Also you will need some way to drive the backlight. My first prototype was a mess and wires kept coming loose when I moved something but at least I proved that my display worked and that I had the right pin outs.

Also you can verify the touch panel works much easier since it only has a few pins. I unsoldered the FPC connector off of the original PCB and soldered on wires that I could connect to an Arduino or CircuitPython board.

After I proved the pin outs worked, I designed a PCB with a backlight driver and all the display connections between the ESP32-S3 development board and the display. That made it much more compact and portable and less likely to come apart during testing.

I can provide more info to get you started, so just feel free to ask questions on what you need clarified.

This GitHub issue is related to CircuitPython specifically so if you have other random questions we may want to move that over to the hackaday.IO page. Either way, just ask and I’ll tell you what I know.

[kmatch98]

I pushed a commit of my code that uses the ESP32-S3 RGB LCD peripheral here: https://github.com/kmatch98/circuitpython/tree/esp32s3_dotclockdisplay

Also, here are my updates to the ESP-IDF lcd peripheral code that support these changes: https://github.com/kmatch98/esp-idf/tree/release/v4.4_rgblcd

[ladyada]

cc @PaintYourDragon

[kmatch98]

As a related issue to "tearing", here is an issue on the esp-idf:

https://github.com/espressif/esp-idf/issues/9121

[kmatch98]

A brief status summary:

The ESP32-S3 LCD peripheral is now proved to be working with CircuitPython. I've got it working with displayio and responding to touch inputs.

https://user-images.githubusercontent.com/33587466/178118126-bb2b849f-c304-4eee-9b3e-8ba48a1f7729.MOV

I pushed my first commit of CircuitPython incorporating the ESP32-S3 dotclockdisplay using the framebufferio. I pushed my version of the ESP-IDF that has some updates to 1) prevent overwhelming the PSRAM DMA access and starving the LCD peripheral 2) change the display init function to allow for a pre-allocated framebuffer and 3) add a function so that CircuitPython display can trigger the display refresh.

To do before merging:

• Request updates to ESP-IDF to allow the RGB display init function to accept a pre-allocated framebuffer
• Rearrange the files so that the dotclockdisplay is only built for the ESP32-S3 port. I think there should be a "notImplemented" version added into the base CP version, but only compile the ESP32-S3 code for selected boards.
• Fix import file directory tree so it will compile without my "hacky" symbolic links.
• Evaluate updates that were merged into the ESP-IDF main branch:
  • Faster drawing in the draw_bitmap function.
    • There seem to be other changes to PSRAM access. Need more information from Espressif if these updates somehow affect the PSRAM DMA memory collisions. If so, the "bounce-buffer" code can be eliminated.
• Other bugs:
  • The display doesn't start updating until about 5 seconds after it seems like it should.
  • When the code finishes and drops into the REPL command, the display goes weird. Is this a dotclockdisplay issue or a framebufferio issue?
  • @FoamyGuy identified that when the screen is not changing, the display is not being refreshed. Perhaps this is because the dirty rectangle tracking does not see a need for a refresh. For the RGB display (that requires periodic refreshes), we should make an option to set a minimum refresh rate that is triggered even when the display I’d not changed.

[tannewt]

• When the code finishes and drops into the REPL command, the display goes weird. Is this a dotclockdisplay issue or a framebufferio issue?

This could be an issue with keeping the framebuffer memory untouched when the MicroPython heap disappears. You'll want to "move" it to the supervisor managed memory.

[kmatch98]

When I init the display, I allocate the FrameBuffer using m_malloc:

    esp_lcd_rgb_panel_config_t panel_config = {
        .fb = m_malloc(fb_size, true), // preallocate framebuffer in CircuitPython

when the code.py finishes, what happens to display objects and FrameBufferIO objects? What do I need to do to ensure that they “live” through after code.py finishes? Also, is m_malloc the right thing to use? @tannewt What do you mean to “move” the FrameBuffer to supervised memory?

(Side Note for posterity: I had to hack the ESP-IDF to accept a prellocated FrameBuffer. The ESP-IDF RGB init function is written to go get a chunk of PSRAM for the FrameBuffer. But CircuitPython pre-grabs all the PSRAM for the heap. So, I modified the RGB display init code to allow to pass in a preallocated FrameBuffer. So in my code, I allocate the FrameBuffer first using m_malloc and then I initialize the display. )

[tannewt]

m_malloc is the correct thing to do but it only applies when the VM is running.

Display objects get copied into static memory here: https://github.com/adafruit/circuitpython/blob/main/shared-module/displayio/__init__.c#L162

Framebuffer memory gets moved here: https://github.com/adafruit/circuitpython/blob/4c20b3cb63abee964c34bd24000a41bad0f53579/supervisor/shared/display.c#L158=

[kmatch98]

I made a first attempt to resolve the issue identified by @FoamyGuy where the screen is not refreshed whenever no changes are drawn to the screen.

I modified this line to force a refresh, even when there are no "dirty" areas: https://github.com/kmatch98/circuitpython/blob/fcde108d03cf589362f681e7e5acf6677e4d7be0/shared-module/framebufferio/FramebufferDisplay.c#L263

I forced this line to True always. It does refresh the screen and prevent it from graying out, but adds some glitching of the screen, particularly at the bottom.

https://user-images.githubusercontent.com/33587466/180249772-9e0de158-f5de-4659-9635-3425e670de1a.MOV

This glitching appears to be only when the screen is "stagnant" with no drawing changes. On a different test with regular changes to the screen (bouncing square) the glitches are absent. I'm currently unsure the cause, perhaps with no changes to the screen the refreshes are being called too frequently (perhaps before the screen is fully redrawn)?

I tried to reduce the refresh rate, but the result is the same:

display.refresh(target_frames_per_second=1, minimum_frames_per_second=1)

[kmatch98]

Foamyguy is making progress on several items in this branch: https://github.com/foamyguy/circuitpython/tree/foamy_tablet

[kmatch98]

Another issue to resolve:

foamyguy noted that the unit is tripping into HardFault whenever the display is initialized and then the code is rerun using Ctrl-D or re-saving code.py. I confirmed this observation by watching the console using the “tio” terminal.

Two possible causes I can think of now. First is mentioned above that the code currently doesn’t move the framebuffer off the heap when the virtual machine closes. Perhaps that is causing something to write into memory out in “lala land” and corrupting the memory.

Second, I don’t think I completed the DotClockDisplay deinit function. Perhaps all the display pins are not being reset properly or the RGB display and LCD peripheral is somehow still “connected” and it faults upon reset.

Separate todo: Also related to pin checking (but likely not causing the HardFault), the current code for the RGB DotClockDisplay doesn’t check if pins are in use before connecting them to the RGB peripheral. Probably need to add pin checking before calling the LCD peripheral init function.

[FoamyGuy]

With a debug build I saw this error printed when the hard fault occured:

Assertion 'heap != NULL && "free() target pointer is outside heap areas"' failed, at file heap_caps.c:339

This was in the normal serial console output (same as REPL). If I understand correctly there may be more, or different output sent over the uart port. I will see what I can find on that one next.

[FoamyGuy]

This is the output from the uart USB port when the hard fault occurs:

hack_tablet_debugging_output_hard_fault.txt

I'm not entirely sure how to read what it has output, but it does have this section which indicates it's a backtrace:

Backtrace:0x420210A9:0x3FCF50700x42021C6F:0x3FCF5090 0x42020DE9:0x3FCF50B0 0x42020EFF:0x3FCF50D0 0x40379AD2:0x3FCF50F0 0x403884D1:0x3FCF5110 0x420A5531:0x3FCF5130 0x420A54C2:0x3FCF5150 0x42049A00:0x3FCF5180 0x42049A0D:0x3FCF51A0 0x42047365:0x3FCF51C0 0x42046325:0x3FCF51E0 0x4203AAEB:0x3FCF5200 0x420132C7:0x3FCF5220 0x4200E63E:0x3FCF5240 0x4200E749:0x3FCF5260 0x4201CAF2:0x3FCF5280 0x420134AA:0x3FCF5320 0x4200E63E:0x3FCF5350 0x4200E662:0x3FCF5370 0x4205481B:0x3FCF5390 0x42054B92:0x3FCF5430 0x42020728:0x3FCF5450 0x42020AA8:0x3FCF5470 0x42020E7E:0x3FCF54E0 0x4202120F:0x3FCF5510 0x42112CED:0x3FCF5530 

There are also these red error outputs in the uart. These seem to appear both during successful executions and when the hard fault occurs though so I think maybe it's not directly related to the hard fault, but not too sure what I'm looking at so included just in case.

E (1290) I2S: i2s_driver_uninstall(2006): I2S port 0 has not installed
E (1290) I2S: i2s_driver_uninstall(2006): I2S port 1 has not installed
E (1290) timer_group: timer_deinit(310): HW TIMER NEVER INIT ERROR
E (1300) timer_group: timer_deinit(310): HW TIMER NEVER INIT ERROR
E (1310) timer_group: timer_deinit(310): HW TIMER NEVER INIT ERROR
E (1310) timer_group: timer_deinit(310): HW TIMER NEVER INIT ERROR

[FoamyGuy]

Here is the decoded version of the backtrace:

❯ python tools/decode_backtrace.py espressif_esp32s3_devkitc_1_n8r8
espressif_esp32s3_devkitc_1_n8r8

0x420210a9: reset_cpu at circuitpython/ports/espressif/supervisor/port.c:315
0x42020de9: __fatal_error at circuitpython/ports/espressif/../../main.c:1001
0x42020eff: __assert_func at circuitpython/ports/espressif/../../main.c:1008
0x40379ad2: heap_caps_free at circuitpython/ports/espressif/build-espressif_esp32s3_devkitc_1_n8r8/esp-idf/../../esp-idf/components/heap/heap_caps.c:339
 (inlined by) heap_caps_free at circuitpython/ports/espressif/build-espressif_esp32s3_devkitc_1_n8r8/esp-idf/../../esp-idf/components/heap/heap_caps.c:324
0x403884d1: free at circuitpython/ports/espressif/build-espressif_esp32s3_devkitc_1_n8r8/esp-idf/../../esp-idf/components/newlib/heap.c:39
0x420a5531: rgb_panel_del at circuitpython/ports/espressif/build-espressif_esp32s3_devkitc_1_n8r8/esp-idf/../../esp-idf/components/esp_lcd/src/esp_lcd_rgb_panel.c:305
0x420a54c2: esp_lcd_panel_del at circuitpython/ports/espressif/build-espressif_esp32s3_devkitc_1_n8r8/esp-idf/../../esp-idf/components/esp_lcd/src/esp_lcd_panel_ops.c:28 (discriminator 2)
0x42049a00: common_hal_dotclockdisplay_framebuffer_deinit at circuitpython/ports/espressif/../../shared-module/dotclockdisplay/DotClockFramebuffer.c:223
0x42049a0d: dotclockdisplay_framebuffer_deinit at circuitpython/ports/espressif/../../shared-module/dotclockdisplay/DotClockFramebuffer.c:239
0x42047365: release_framebufferdisplay at circuitpython/ports/espressif/../../shared-module/framebufferio/FramebufferDisplay.c:370
0x42046325: common_hal_displayio_release_displays at circuitpython/ports/espressif/../../shared-module/displayio/__init__.c:126
0x4203aaeb: displayio_release_displays at circuitpython/ports/espressif/../../shared-bindings/displayio/__init__.c:72
0x420132c7: fun_builtin_0_call at circuitpython/ports/espressif/../../py/objfun.c:58
0x4200e63e: mp_call_function_n_kw at circuitpython/ports/espressif/../../py/runtime.c:665
0x4200e749: mp_call_method_n_kw at circuitpython/ports/espressif/../../py/runtime.c:680
0x4201caf2: mp_execute_bytecode at circuitpython/ports/espressif/../../py/vm.c:1021
0x420134aa: fun_bc_call at circuitpython/ports/espressif/../../py/objfun.c:297 (discriminator 4)
0x4200e63e: mp_call_function_n_kw at circuitpython/ports/espressif/../../py/runtime.c:665
0x4200e662: mp_call_function_0 at circuitpython/ports/espressif/../../py/runtime.c:638
0x4205481b: parse_compile_execute at circuitpython/ports/espressif/../../shared/runtime/pyexec.c:146
0x42054b92: pyexec_file at circuitpython/ports/espressif/../../shared/runtime/pyexec.c:745
0x42020728: maybe_run_list at circuitpython/ports/espressif/../../main.c:227
0x42020aa8: run_code_py at circuitpython/ports/espressif/../../main.c:403
0x42020e7e: main at circuitpython/ports/espressif/../../main.c:934
0x4202120f: app_main at circuitpython/ports/espressif/supervisor/port.c:412
0x42112ced: main_task at circuitpython/ports/espressif/build-espressif_esp32s3_devkitc_1_n8r8/esp-idf/../../esp-idf/components/freertos/port/port_common.c:141

(edited to remove a million duplicates of path strings that are likely not relevant)

[kmatch98]

I have no experience reading this debug trace, but at first glance it looks like the de_init of the display is failing. I assume that heap_caps_free is trying to free the FrameBuffer memory. Due to the way I implemented the FrameBuffer malloc it’s possible that the ESP-IDF can’t properly free the memory. (The ESP-IDF display init code assumes it can malloc the PSRAM for the FrameBuffer. In the case of CircuitPython, CP grabs all the PSRAM as it’s heap and thus leaves nothing for the ESP-IDF to access. I hacked the ESP-IDF to get around it, but @jepler came up with a clean way around this.). It’s possible that taking advantage of @jepler ’s recent PR to allocate some separate heap to be managed by the ESP-IDF could resolve this. Even if it doesn’t solve this specific problem, it’s probably a good idea to use the new PR capability, since it will reduce the changes required to the ESP-IDF code.

[RetiredWizard]

I've been playing with this trying to get @kmatch98 's and @FoamyGuy 's repos merged with CP 8.0.0 beta.

Maybe someone can help with my current stumbling block. esp-idf/components/esp_lcd/src/esp_lcd_rgb_panel.c references hal/lcd_ll.h but I can't seem to locate that header file. The fact that I can't find that file has me worried I've missed something basic.

I aslo threw these two lines:

 #define SOC_LCD_RGB_SUPPORTED       1
 #define SOC_LCD_RGB_DATA_WIDTH      16

in ports/espressif/boards/espressif_esp32s2_devkitc_1_n4r2/mpconfigboard.h (the board I've been building torwards), and I'm not sure that was the correct place or the right way to deal with the missing parameters.

I guess I'll ask this as well, although I had planned on digging harder into it once I was able to build but I haven't spotted yet where the data pins are defined. I'm assuming the EXAMPLE_PIN_NUMBER variables are not the correct place.

:-)

[RetiredWizard]

Well progress on one front, when all else fails start reading the documentation:

From esp-idf/components/hal/readme.md:

"often somtimes xxx_hal.h includes a lower-level header from a xxx_ll.h, which resides in the implementation."

And I can see I'll have to change my target board, kmatch98's code is targeting the LCD controller on the S3 :/

[eidetech]

@kmatch98: Probably not allowed to ask this question here, but I could not find any other way to get in contact with you.

I see that you are running your screen at 10MHz, and still on the video it looks like you are not experiencing any flickering.

I have a RGB LCD that I am trying to use with a ESP32-S3 16MB Flash, 2MB PSRAM, but I am experiencing visible flickering, especially when running at low DCLK. Do you have any tips or tricks to get rid of it?

[kmatch98]

It’s been a while since I worked on this so I am rusty on some of the details. From what I remember unfortunately there are many different ways to cause flickering.

May be helpful if you add a video so folks here might have ideas. Also, are you actually running CircuitPython or are you just using the ESP-IDF example code? I do know there are quite a few LCD display based peripheral code updates in the latest version of the ESP-IDF, so whatever observations you have may be affected by which version you are running.

One thing I struggled with was the conflict of accessing the PSRAM bus, where using DMA for the LCD peripheral conflicted with other processes accessing the PSRAM and they’re wasn’t enough bandwidth to keep up with the display refresh rate. For that specific issue, the ESP-IDF team added something called a “bounce buffer” that in essence gives the LCD peripheral all the bandwidth (and actually defeats the purpose of DMA access). The ESP32 forum may have some useful threads too if you search on LCD or RGB display.

Anyway, if you post more detailed info here maybe someone can point you in the right direction.

[kmatch98]

One more thing I remember is that with using a single buffer I was getting tearing when updating the buffer (depending upon what you are displaying, this may appear as flickering). The original version of the ESP-IDF did not have a way of preventing tearing, the display was set to just automatically redraw at the fastest rate possible. I think it was designed to trigger a redraw interrupt after the screen was redrawn.

I had to turn off the interrupt and change the trigger for redrawing so I could actively request a redraw from my code. I think this option may have been included in the latest ESP-IDF version.

[eidetech]

I'm using the ESP-IDF RGB Panel example code, not anything CircuitPython related. I have been using the latest esp-idf 5.0. Also, I have been using double buffer.

The screen drawing is working fine, and the flickering is also present when no new drawing is done, meaning the screen is static. The flickering appears as "visible lines" going vertically down the screen, like old LCD TV's.

I took a video of it the other day: https://www.youtube.com/watch?v=7I9NJPrJ3JY

[kmatch98]

Actually I think mine shows a little bit of what you show in the video. Not sure what causes it.

I recommend you go look at the code that triggers a refresh and hack it to see if you can “manually” control the redraw time. Then you can actively change the redraw rate and confirm whether the redraw is what is affecting your flickering.

Sorry I don’t have more insights.

[eidetech]

Ok, thanks for the insights @kmatch98.

[Syrinx98]

@kmatch98 hi, i've been working recently on esp32s3-wroom1-n16r8 and rgb lcd, i just read all the messages here and i would like to ask you if you can help me understand here what's happening with my code. I have two different TFTs, one is 7 inch (800 x 480) the other is 10 inch (1024 x 600) the only way i found to make the screens go decent without errors or so is using double buffers, but to maintain synchronisation with the two buffers, i need to set the "full refresh" to true.

I paste here my lcd.h and lcd.c

lcd.h

#pragma once

#include "lvgl.h"
#include "utils.h"
#include "esp_timer.h"
#include "esp_lcd_panel_ops.h"
#include "esp_lcd_panel_rgb.h"
#include "esp_lcd_touch.h"
#include "esp_lcd_touch_gt911.h"
#include "esp_lvgl_port.h"

// SE DOBBIAMO AGGIORNARE GRANDI QUANTITA' DI PIXEL, DOUBLE BUFFER E' PERFETTO PERCHE' NON VEDO MALE NIENTE MA HO POCHI FPS DI DEFAULT
// SE DOBBIAMO AGGIORNARE PICCOLE QUANTITA' DI PIXEL E LA SCHERMATA E' PER LA MAGGIOR PARTE STATICA, ALLORA UTILIZZIAMO OBBLIGATORIAMENTE DUE BOUNCE BUFFER (CON QUESTA MODALITà POSSIAMO ALZARE MHZ)

// se CONFIG_EXAMPLE_DOUBLE_FB è 1 sto usando un double frame buffer, altrimenti sto utilizzando il bounce buffer
#define CONFIG_EXAMPLE_DOUBLE_FB                                1
#define CONFIG_EXAMPLE_AVOID_TEAR_EFFECT_WITH_SEM               0

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
//////////////////// Please update the following configuration according to your LCD spec //////////////////////////////
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
#define SETTE_POLLICI 0
#define DIECI_POLLICI 1

#define SELECTED_TFT DIECI_POLLICI

#if SELECTED_TFT == SETTE_POLLICI

// TODO VEDERE COME VA CON IL SETTE POLLICI BOUNCE BUFFER
#define EXAMPLE_LCD_PIXEL_CLOCK_HZ     (16 * 1000 * 1000)
#define EXAMPLE_LCD_H_RES              800
#define EXAMPLE_LCD_V_RES              480
#define HSYNC_BACK_PORCH 43
#define HSYNC_FRONT_PORCH 8
#define HSYNC_PULSE_WIDTH 4
#define VSYNC_BACK_PORCH 12
#define VSYNC_FRONT_PORCH 37
#define VSYNC_PULSE_WIDTH 37

#define MS_BETWEEN_FRAMES 30

#elif SELECTED_TFT == DIECI_POLLICI

#if CONFIG_EXAMPLE_DOUBLE_FB
#define EXAMPLE_LCD_PIXEL_CLOCK_HZ     (16 * 1000 * 1000)
#else
#define EXAMPLE_LCD_PIXEL_CLOCK_HZ     (30 * 1000 * 1000)
#endif

#define EXAMPLE_LCD_H_RES              1024
#define EXAMPLE_LCD_V_RES              600
#define HSYNC_BACK_PORCH 160
#define HSYNC_FRONT_PORCH 16
#define HSYNC_PULSE_WIDTH 1
#define VSYNC_BACK_PORCH 23
#define VSYNC_FRONT_PORCH 12
#define VSYNC_PULSE_WIDTH 1

#if CONFIG_EXAMPLE_DOUBLE_FB
#define MS_BETWEEN_FRAMES 38
#else
#define MS_BETWEEN_FRAMES 16 // lo schermo può andare a 60 fps a livello teorico
#endif
#endif

#define EXAMPLE_PIN_NUM_HSYNC          6
#define EXAMPLE_PIN_NUM_VSYNC          7
#define EXAMPLE_PIN_NUM_DE             15
#define EXAMPLE_PIN_NUM_PCLK           21

#define EXAMPLE_PIN_NUM_DATA0          42 // B3
#define EXAMPLE_PIN_NUM_DATA1          44 // B4
#define EXAMPLE_PIN_NUM_DATA2          43 // B5
#define EXAMPLE_PIN_NUM_DATA3          2 // B6
#define EXAMPLE_PIN_NUM_DATA4          1 // B7

#define EXAMPLE_PIN_NUM_DATA5          14 // G2
#define EXAMPLE_PIN_NUM_DATA6          46 // G3
#define EXAMPLE_PIN_NUM_DATA7          38 // G4
#define EXAMPLE_PIN_NUM_DATA8          39 // G5
#define EXAMPLE_PIN_NUM_DATA9          40 // G6
#define EXAMPLE_PIN_NUM_DATA10         41 // G7

#define EXAMPLE_PIN_NUM_DATA11         47  // R3
#define EXAMPLE_PIN_NUM_DATA12         48  // R4
#define EXAMPLE_PIN_NUM_DATA13         45 // R5
#define EXAMPLE_PIN_NUM_DATA14         0 // R6
#define EXAMPLE_PIN_NUM_DATA15         17 // R7

#define EXAMPLE_PIN_NUM_DISP_EN        -1

#define EXAMPLE_LVGL_TICK_PERIOD_MS    1

extern pthread_mutex_t lvgl_mutex;

void increase_lvgl_tick(void *arg);
void example_lvgl_flush_cb(lv_disp_drv_t *drv, const lv_area_t *area, lv_color_t *color_map);
bool example_on_vsync_event(esp_lcd_panel_handle_t panel, const esp_lcd_rgb_panel_event_data_t *event_data, void *user_data);

void lcd_init();

void lvgl_task(void *pvParameters);

lcd.c

#include "lcd.h"
#include "i2c.h"

static const char *TAG = "LCD SCREEN";

#include "driver/gpio.h"
#include "utils.h"
#include "esp_system.h"

#include "esp_heap_caps.h"
#include "esp_system.h"

void *buf1 = NULL;
void *buf2 = NULL;

pthread_mutex_t lvgl_mutex;

// we use two semaphores to sync the VSYNC event and the LVGL task, to avoid potential tearing effect
#if CONFIG_EXAMPLE_AVOID_TEAR_EFFECT_WITH_SEM
SemaphoreHandle_t sem_vsync_end;
SemaphoreHandle_t sem_gui_ready;
#endif

bool example_on_vsync_event(esp_lcd_panel_handle_t panel, const esp_lcd_rgb_panel_event_data_t *event_data, void *user_data)
{
    BaseType_t high_task_awoken = pdFALSE;
#if CONFIG_EXAMPLE_AVOID_TEAR_EFFECT_WITH_SEM
    if (xSemaphoreTakeFromISR(sem_gui_ready, &high_task_awoken) == pdTRUE) {
        xSemaphoreGiveFromISR(sem_vsync_end, &high_task_awoken);
    }
#endif
    return high_task_awoken == pdTRUE;
}

void example_lvgl_flush_cb(lv_disp_drv_t *drv, const lv_area_t *area, lv_color_t *color_map)
{

    esp_lcd_panel_handle_t panel_handle = (esp_lcd_panel_handle_t) drv->user_data;
    int offsetx1 = area->x1;
    int offsetx2 = area->x2;
    int offsety1 = area->y1;
    int offsety2 = area->y2;

#if CONFIG_EXAMPLE_AVOID_TEAR_EFFECT_WITH_SEM
    xSemaphoreGive(sem_gui_ready);
    xSemaphoreTake(sem_vsync_end, portMAX_DELAY);
#endif

    esp_lcd_panel_draw_bitmap(panel_handle, offsetx1, offsety1, offsetx2 + 1, offsety2 + 1, color_map);
    lv_disp_flush_ready(drv); 
}

void increase_lvgl_tick(void *arg)
{
    /* Tell LVGL how many milliseconds has elapsed */
    lv_tick_inc(EXAMPLE_LVGL_TICK_PERIOD_MS);
}

void lcd_init()
{

    static lv_disp_draw_buf_t disp_buf; // contains internal graphic buffer(s) called draw buffer(s)
    static lv_disp_drv_t disp_drv; // contains callback functions

#if CONFIG_EXAMPLE_AVOID_TEAR_EFFECT_WITH_SEM
    ESP_LOGI(TAG, "Create semaphores");
    sem_vsync_end = xSemaphoreCreateBinary();
    assert(sem_vsync_end);
    sem_gui_ready = xSemaphoreCreateBinary();
    assert(sem_gui_ready);
#endif

    ESP_LOGI(TAG, "Install RGB LCD panel driver");
    esp_lcd_panel_handle_t panel_handle = NULL;
    esp_lcd_rgb_panel_config_t panel_config = {
        .data_width = 16,
        .psram_trans_align = 64,
        .clk_src = LCD_CLK_SRC_PLL240M,
#if CONFIG_EXAMPLE_DOUBLE_FB == 0
//.bounce_buffer_size_px = 20 * EXAMPLE_LCD_H_RES,
#endif
        .disp_gpio_num = EXAMPLE_PIN_NUM_DISP_EN,
        .pclk_gpio_num = EXAMPLE_PIN_NUM_PCLK,
        .vsync_gpio_num = EXAMPLE_PIN_NUM_VSYNC,
        .hsync_gpio_num = EXAMPLE_PIN_NUM_HSYNC,
        .de_gpio_num = EXAMPLE_PIN_NUM_DE,
        .data_gpio_nums = {
        EXAMPLE_PIN_NUM_DATA0,
        EXAMPLE_PIN_NUM_DATA1,
        EXAMPLE_PIN_NUM_DATA2,
        EXAMPLE_PIN_NUM_DATA3,
        EXAMPLE_PIN_NUM_DATA4,
        EXAMPLE_PIN_NUM_DATA5,
        EXAMPLE_PIN_NUM_DATA6,
        EXAMPLE_PIN_NUM_DATA7,
        EXAMPLE_PIN_NUM_DATA8,
        EXAMPLE_PIN_NUM_DATA9,
        EXAMPLE_PIN_NUM_DATA10,
        EXAMPLE_PIN_NUM_DATA11,
        EXAMPLE_PIN_NUM_DATA12,
        EXAMPLE_PIN_NUM_DATA13,
        EXAMPLE_PIN_NUM_DATA14,
        EXAMPLE_PIN_NUM_DATA15,
    },
        .timings = {
        .pclk_hz = EXAMPLE_LCD_PIXEL_CLOCK_HZ,
        .h_res = EXAMPLE_LCD_H_RES,
        .v_res = EXAMPLE_LCD_V_RES,
        // The following parameters should refer to LCD spec
        .hsync_back_porch = HSYNC_BACK_PORCH,
        .hsync_front_porch = HSYNC_FRONT_PORCH,
        .hsync_pulse_width = HSYNC_PULSE_WIDTH,

        .vsync_back_porch = VSYNC_BACK_PORCH,
        .vsync_front_porch = VSYNC_FRONT_PORCH,
        .vsync_pulse_width = VSYNC_PULSE_WIDTH, 
    },

        .flags.fb_in_psram = true,
#if CONFIG_EXAMPLE_DOUBLE_FB
        .flags.double_fb = true,
#else
        .flags.double_fb = false,
#endif 

    };
    ESP_ERROR_CHECK(esp_lcd_new_rgb_panel(&panel_config, &panel_handle));

    ESP_LOGI(TAG, "Register event callbacks");
    esp_lcd_rgb_panel_event_callbacks_t cbs = {
        .on_vsync = example_on_vsync_event,
    };
    ESP_ERROR_CHECK(esp_lcd_rgb_panel_register_event_callbacks(panel_handle, &cbs, &disp_drv));

    ESP_LOGI(TAG, "Initialize RGB LCD panel");
    ESP_ERROR_CHECK(esp_lcd_panel_reset(panel_handle));
    ESP_ERROR_CHECK(esp_lcd_panel_init(panel_handle));

    // LIVELLO LVGL IN POI, DUNQUE CONFIGURO IL TOUCH
    ESP_LOGI(TAG, "Initialize LVGL library");
    lv_init();

    ESP_LOGI(TAG, "Allocate separate LVGL draw buffers from PSRAM");

    /*

    */

#if CONFIG_EXAMPLE_DOUBLE_FB
    ESP_LOGI(TAG, "Use frame buffers as LVGL draw buffers");

    ESP_ERROR_CHECK(esp_lcd_rgb_panel_get_frame_buffer(panel_handle, 2, &buf1, &buf2));

    // initialize LVGL draw buffers
    lv_disp_draw_buf_init(&disp_buf, buf1, buf2, EXAMPLE_LCD_H_RES * EXAMPLE_LCD_V_RES);
#else
    buf1 = heap_caps_malloc(EXAMPLE_LCD_H_RES * 100 * sizeof(lv_color_t), MALLOC_CAP_DMA | MALLOC_CAP_32BIT);
    assert(buf1);
    // initialize LVGL draw buffers
    lv_disp_draw_buf_init(&disp_buf, buf1, NULL, EXAMPLE_LCD_H_RES * 100);
#endif

    ESP_LOGI(TAG, "Register display driver to LVGL");
    lv_disp_drv_init(&disp_drv);
    disp_drv.hor_res = EXAMPLE_LCD_H_RES;
    disp_drv.ver_res = EXAMPLE_LCD_V_RES;
    disp_drv.flush_cb = example_lvgl_flush_cb;
    disp_drv.draw_buf = &disp_buf;
    disp_drv.user_data = panel_handle;
#if CONFIG_EXAMPLE_DOUBLE_FB
    disp_drv.direct_mode = true;
    disp_drv.full_refresh = true; // the full_refresh mode can maintain the synchronization between the two frame buffers
#else
    disp_drv.direct_mode = false;
    disp_drv.full_refresh = false; 
#endif
    lv_disp_t *disp = lv_disp_drv_register(&disp_drv);

#if SELECTED_TFT == DIECI_POLLICI
    esp_lcd_touch_handle_t tp;

    /* Initialize touch */
    const esp_lcd_touch_config_t tp_cfg = {
        .x_max = EXAMPLE_LCD_H_RES,
        .y_max = EXAMPLE_LCD_V_RES,
        .rst_gpio_num = -1,
        .int_gpio_num = -1,
        .levels = {
        .reset = 0,
        .interrupt = 0,
    },
        .flags = {
        .swap_xy = 0,
        .mirror_x = 0,
        .mirror_y = 0,
    },
    };
    esp_lcd_panel_io_handle_t tp_io_handle = NULL;
    const esp_lcd_panel_io_i2c_config_t tp_io_config = ESP_LCD_TOUCH_IO_I2C_GT911_CONFIG();
    ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c((esp_lcd_i2c_bus_handle_t)I2C_NUM_0, &tp_io_config, &tp_io_handle));
    ESP_ERROR_CHECK(esp_lcd_touch_new_i2c_gt911(tp_io_handle, &tp_cfg, &tp));
    assert(tp);

    /* Add touch input (for selected screen) */
    const lvgl_port_touch_cfg_t touch_cfg = {
        .disp = disp,
        .handle = tp,
    };

    lvgl_port_add_touch(&touch_cfg);

#endif

    ESP_LOGI(TAG, "Install LVGL tick timer");
    // Tick interface for LVGL (using esp_timer to generate 2ms periodic event)
    const esp_timer_create_args_t lvgl_tick_timer_args = {
        .callback = &increase_lvgl_tick,
        .name = "lvgl_tick"
    };
    esp_timer_handle_t lvgl_tick_timer = NULL;
    ESP_ERROR_CHECK(esp_timer_create(&lvgl_tick_timer_args, &lvgl_tick_timer));
    ESP_ERROR_CHECK(esp_timer_start_periodic(lvgl_tick_timer, EXAMPLE_LVGL_TICK_PERIOD_MS * 1000));

    //ESP_LOGI(TAG, "Display LVGL Scatter Chart");
}

void lvgl_task(void *pvParameters) {
    for (;;) {
        pthread_mutex_lock(&lvgl_mutex);
        lv_timer_handler();
        pthread_mutex_unlock(&lvgl_mutex);
        // raise the task priority of LVGL and/or reduce the handler period can improve the performance
        vTaskDelay(pdMS_TO_TICKS(MS_BETWEEN_FRAMES));
        // The task running lv_timer_handler should have lower priority than that running `lv_tick_inc`

    }
}

as you can see from my code, i followed the steps given by the docs, i can obtan the result you can see on the video (this is the 7inch, but the 10inch works similary). You can see from the video that is it 33 fps, but i think when i use double frame buffer, lvgl fails to properly calculate the fps. Also, you can see from the spinners on the right, that the screen is smooth enough, but i don't know if i'm doing something wrong with my code or maybe something could be optimized to try to get more FPS (at the moment i have to fully refresh the screen to maintain sync between the 2 fb, but maybe a it is not the only way to do so, i would like to be able to not refresh the whole screen for every time, because if in another screen i have only 1 widget, it goes the same fps as the one you see on the video, because it simply refreshes all every time).

I also tried with single frame buffer and bounce buffer, but whenever i have a lot of pixels to update, i sometimes have a desync effect for graphics that are vertically long.

Thank you in advance for your help and your time.

https://user-images.githubusercontent.com/71718486/231767421-eb1522c9-4544-4152-8a6b-d8db4489f1e1.mp4

[kmatch98]

This looks like great work. As for the frame per second. Based on your dot clock and all the timings, if you calculate the max possible frame rate, does it match to what LVGL is saying. From what I remember (it’s been a while since I worked on this), I think getting 30 fps was really at the high end. I suspect you can’t achieve this for the larger display but you can calculate your best case based on your settings and the number of pixels.

If you redraw the full framebuffer each time, I think this is going to be your limit. And keep in mind with dot clock displays the display itself doesn’t have any real “memory” other than the discharge time of each pixel.

For displays that have a separate display controller with memory, you can sometimes speed up the fps by only writing the pixels that changed.

For dot clock displays I’m unsure if you have that flexibility. Maybe you could run a test case where you try to only redraw a small portion of the display and see how the display behaves.

Sorry I don’t have exact answers for you. I think that getting a smooth running display is a good challenge. Maybe once you calculate your maximum possible frame rate you can see how close you are to that and then decide some next steps.

[Syrinx98]

Thank you for your reply.

After some tests i can say that:

• if i have to redraw little portions of screen but i use the double frame buffer, nothing changes (as expected)

• if i have to redraw little portions of screen but i use singla frame buffer + bounce buffer, i also can set the lcd clk at higher speed so i can get 50 fps (that would be the case of some static interfaces with little widgets and no animations).

• i am going to 80Mhz with octal PSRam in DDR and the DMA is automatically configured setting ".psram_trans_align", i should be able to go at 160 MB/s.

• i asked in the esp forum they say that the problem is the "arbiter" component I don't really know what to do right now, in the v5.1 they make the s3 able to set 120Mhz in octal mode psram, i could wait for that release...

[Syrinx98]

Also, can you please explain if i have to use esp-idf "draw_bitmap" or the init of the lcd screen, because i saw from the other messages here that you override those and you used your own, do you think that it could solve my problem?

[kmatch98]

I overrode some of those functions because an earlier revision of the esp-idf didn’t support some of the functions that I needed. I think the latest version has added all of those. So At first glance I don’t suspect that is affecting your performance. It may be primarily limited by the PSRAM access speeds.

As for “arbiter” I’m haven’t heard that term so unfortunately can’t provide any new insight. Keep digging, sorry I can’t be more helpful.