esp_lcd RGB panel buffer issue. (IDFGH-12838)

kdschlosser commented 3 months ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

5.0.6

Espressif SoC revision.

ESP32-S3 N16R8

Operating System used.

Linux

How did you build your project?

Command line with CMake

If you are using Windows, please specify command line type.

None

Development Kit.

ESP32-S3 Devkit

Power Supply used.

USB

What is the expected behavior?

The display to render properly

What is the actual behavior?

I am not sure what you would call it, It resembles tearing but I do not think that is what is happening.

Here is a small video that shows the issue.

glitch

I am calling lv_display_flush_ready from inside of the vsync callback. I have tried using the semaphores like what is done in the example and there was no change at all. It is almost like the buffer is being written to while it is sending.

This is as much a question as it is a possible bug. I was looking at the source code in the ESP-IDF and I am not sure exactly when and how the buffers get flip flopped. Is the swapping of the buffers the responsibility of the user? Or is it handled by the driver? Is it the responsibility of the user to keep things in sync and only call esp_lcd_panel_draw_bitmap during a vsync event?.

I am using octal SPIRAM and octal FLASH. should I push the transfer speed up to 120mhz? and if so how do I got about doing that?

Steps to reproduce.

RGB bus display Lanes: 16 Width: 800 Height: 480 Color format: RGB565 GUI Framework: LVGL 9.1 Number of frame buffers: 2 Bounce buffer: no On-demand: no

Set up the display Create 2 sliders, resize them so they are large, align the sliders to the side of the display loop setting the value to move the slider position

Debug Logs.

No response

More Information.

No response

Lzw655 commented 3 months ago

Hi @kdschlosser, the flickering of the right-side widget in the video is indeed abnormal. This phenomenon is likely not caused by screen tearing. Could you please tell me which model of LCD IC you are using? And could you also upload your project?

kdschlosser commented 3 months ago

Ohh.. the project is quite involved. The project binds LVGL to MicroPython. You can view the binding code here...

https://github.com/kdschlosser/lvgl_micropython/blob/main/ext_mod/lcd_bus/esp32_src/rgb_bus.c

The IC that is being used is the ST7262.

I believe the display being used is a 5" Sunton display board. with the ESP32-S3 onboard.

What it almost looks like to me is a synchronization issue between LVGL and the RGBPanel driver in the esp-idf. When does the panel driver start sending the frame buffer? Does it do it right after initialization? or so it sit there idle until it is told to transfer the buffer?

It looks like the buffer is being written to while it is transferring.....

LVGL has mechanics in it that causes the program to stall while a framebuffer is being transmitted. The issue with RGB is that when the buffer is passed to be transmitted it is always going to be in a state where it is being transmitted. It is like this until the panel driver is told to use a different buffer. When does the panel driver change the buffers? does it wait until a vsync interrupt occurs? or does it change them immediately?

There is no way to really know if the buffer is able to be swapped other than using the vsync event and we don't want to cause a stall in an ISR for obvious reasons. And no memory allocation can occur inside the ISR either. The only thing that is able to be done is we can tell LVGL that the buffer has finished transferring. so what happens when that is done is LVGL will then go about copying the data from the buffer that was just transferred to the second buffer and this will happen at the same time the transfer to the display is occurring. Using the vsync is not a fantastic way to let LVGL know that the buffer has been transferred because LVGL is expecting the buffer to no longer be in use anymore. This is not the case when using the RGB driver. The buffer will always be in a state of being transferred until the panel driver is passed a new buffer.

What throws things out of sync is when we tell the panel driver to write a new buffer and as soon as the next vsync occurs that flag gets set to tell LVGL the buffer is good to go when it actually has not been written yet. LVGL then swaps the buffers around and it now starts writing to the buffer that is in the process of being sent.

I am almost 100% certain that this is being caused by a synchronization issue between the panel driver and LVGL. I am not sure how to fix it without using some kind of an intermediate buffer and it's contents needing to be copied. That is what I am trying to avoid because it's a performance killer when rendering an 800 x 480 x 16bit display.

I did want to note that only a single core is being used, so it's not an issue being caused by external code to the IDF trying to write to the buffers at the same time. DMA memory is being used for the frame buffers which would allow for a bumping of heads between the panel driver and LVGL a problem cannot exist where portions of LVGL code are writing to the frame buffer at the same time.

I have spoken to @kisvegabor on a video chat about this issue and he might be better to explain the mechanics of LVGL.

Lzw655 commented 3 months ago

OK... BTW, we have an example rgb_lcd_8bit that achieves synchronization between LVGL rendering and RGB refreshing through multiple buffers, preventing screen tearing.

kdschlosser commented 3 months ago

@Lzw655

While that is a great example it is self defeating. The design of LVGL and using double buffering with DMA memory is the ability to do other things while the buffer is being transferred. What your code does is it negates that from happening. you are stalling the flush callback until the vsync even occurs which is only going to occur once the buffer has transferred. While it would work to keep things in sync it is not going to provide the best in terms of performance.

We want to be able to do other things while that buffer is transferring and not block the flush callback.

That's where the issue is. how do we do that while keeping things in sync? What I think is happening is in the vsync even which can happen at any time is when we need to let lvgl know the buffer has been transferred but the issue is not knowing what buffer has actually been transferred. if say buffer 1 is in the middle of sending and then LVGL goes and renders new data to the second buffer and then calls the flush function to send the buffer to the display soon after the first buffer finishes and the vsync event gets called. But the second buffer has not actually finished transferring and LVGL has just been notified that it has. See where the problem is? Then LVGL goes to copy information from one buffer to the next in order to sync the buffers and you end up with the problem seen in the example above.

Maybe this issue needs to get corrected in LVGL by adding a flag for each buffer.

I am going to talk to @kisvegabor about this and see what he says.

kdschlosser commented 3 months ago

I found a solution. It's not exactly what I want but at least it is going to allow more things to get processed and done while the framebuffer is transmitting.

Thank you @Lzw655 's your example is what inspired what I did. The problem I had with your example is the program stalling right after calling the tx_color function. Doing that completely negates any performance gains from using double buffering and DMA memory. While I know that double buffering is used to stop issues where rendering to the buffer is being done at the same time the buffer is transmitting it also carries with it a performance gain due to the use of DMA memory and being able to do other things while the transmitting is happening.

So what I did was instead of having the stall right after the tx_color is called in the flush callback I registered another callback with LVGL that gets called at the start of a refresh before anything gets written to the frame buffer.

void lv_display_set_flush_wait_cb(lv_display_t * disp, lv_display_flush_wait_cb_t wait_cb)

So now the flush function will exit and LVGL and do it's thing with timers or whatever else it needs to do and then the user code will run and make any changes that need to be made to LVGL and once lv_task_handler gets called and the timer has expired for the refresh is when that callback will get called. In that callback is where I have the code to collect the notification from FreeRTOS and then I call flush_ready in lvgl which will skip the spinning wheels that is done to wait for that function to get called.

This is going to provide more time to do other things.

Now.. I could shrink the time it spends waiting down even more if there was a function in the ESP-IDF that would allow me to get the frame buffer number by passing the buffer to that function. There is the possibility of a vsync occurring before the code comes full circle. Because there is no way to identify what buffer just flushed in the vsync callback I am not able to notify the task if it needs to wait or if it doesn't.

I need to be able to pull the index number of the buffer that the vsync callback was called for.

kdschlosser commented 3 months ago

another note on this. I submitted a PR to LVGL that reworks the refreshing of the display. If it gets merged there is going to be an option that can be turned on that will kick out of the refresh if there is a buffer that is still being transmitted. This will allow things to continue running if DMA memory is being used instead of sitting there spinning it's wheels. I believe this is going to be a better design as the user can use a non blocking call to check and see if a vsync has occurred and if it has then it can call the flush ready function in the task loop.

espressif / esp-idf