TFT Update Fails if Bluetooth Proxy is enabled `Bug`

[M0uC0]

TFT Version

4.2.6

ESPHome Version

4.2.6

Blueprint Version

4.2.6

Panel Model

EU

What is the bug?

TFT Upload Fails when Bluetooth proxy is enabled

Steps to Reproduce

Updated to 4.2.6 (Blueprint and ESP32) when trying to update TFT it fails. At max speed gets to around 19% and panel reboots At 115200 gets to around 3% and panel reboots

Removed bluetooth_proxy: Reflashed ESP32 with v4.2.6.

No problems uploading TFT at max speed... gets to 100% without any errors or rebooting.

Your Panel's YAML

substitutions:
  # Settings - Editable values
  device_name: "nsp-quarto" 
  wifi_ssid: !secret wifi_ssid
  wifi_password: !secret wifi_password
  nextion_update_url: "http://192.168.99.10:8123/local/nspanel_eu.tft"  # Optional for `esp-idf` framework
  # Add-on configuration (if needed)
  # heater_relay: "1"  # Possible values: "1" or "2"

# Customization area
##### My customization - Start #####

wifi:
  networks:
    - id: !extend wifi_default
    - ssid: !secret wifi_ssid2
      password: !secret wifi_password
  power_save_mode: LIGHT
# Set Wi-Fi power save mode to "LIGHT" as required for Bluetooth on ESP32

# Encrypt the communication between ESPHome and Home Assistant
api:
  encryption:
    key: !secret api_key
  reboot_timeout: 60min

# Use my global OTA password
ota:
  password: !secret ota_key

# Enable Bluetooth proxy
bluetooth_proxy:

##### My customization - End #####

# Core and optional configurations
packages:
  remote_package:
    url: https://github.com/Blackymas/NSPanel_HA_Blueprint
    ref: main
    files:
      - nspanel_esphome.yaml # Core package
      # Optional advanced and add-on configurations
      # - advanced/esphome/nspanel_esphome_advanced.yaml
      # - nspanel_esphome_addon_climate_cool.yaml
      # - nspanel_esphome_addon_climate_heat.yaml
      # - nspanel_esphome_addon_climate_dual.yaml
    refresh: 300s

esp32:
  framework:
    type: esp-idf

ESPHome Logs

No response

Home Assistant Logs

No response

[edwardtfn]

I cannot try this right now, but this should stop the BLE scan before starting the upload:

bluetooth_proxy:
  id: ble_proxy

esp32_ble_tracker:
  id: ble_tracker

script:
  - id: !extend upload_tft
    then:
      - lambda: |-
          static const char *const TAG = "CUSTOM.script.upload_tft";
          ble_tracker->dump_config();
          ESP_LOGD(TAG, "Stopping BLE Tracker scan...");
          ble_tracker->stop_scan();
          ESP_LOGD(TAG, "Disabling BLE Tracker scan...");
          ble_tracker->set_scan_active(false);
          ESP_LOGD(TAG, "State: %s", id(ble_proxy)->has_active() ? "Active" : "Passive");
          while (ble_proxy->get_bluetooth_connections_limit() != ble_proxy->get_bluetooth_connections_free()) {
            ESP_LOGD(TAG, "Connections: %i of %i", int(ble_proxy->get_bluetooth_connections_limit() - ble_proxy->get_bluetooth_connections_free()), int(ble_proxy->get_bluetooth_connections_limit()));
            if (id(ble_proxy)->has_active()) {
              ESP_LOGD(TAG, "Setting passive mode...");
              ble_proxy->set_active(false);
            }
            vTaskDelay(pdMS_TO_TICKS(1000));
            App.feed_wdt();
          }

Could you please let me know if that helps (and please provide logs)?

[edwardtfn]

At max speed gets to around 19% and panel reboots

This sounds like lack of memory. Could you please see the free heap informed during the upload process (before the change I've proposed earlier)?

[M0uC0]

It was only after i made this post that i saw the comments around the "Bluetooth Proxy Customization" in the documentation. I thought that this was new problem because everything worked ok when i updated to 4.2.4. Although i hadn't read the sugestions... i tried the 1st one.. and it worked flawlessly.

I have 2 new panels to install in the next days... i'll flash one with bluetooth proxy enabled and see how it will work when updating the tft... and i'll collect the logs.

[edwardtfn]

i tried the 1st one.. and it worked flawlessly.

Which one was that?

[edwardtfn]

I've improved docs to suggest stopping the BLE scan before upload TFT, but we also solved a memory leak that could be affecting this:

• 1832

I will close this for now to make the backlog a bit more clear, but please feel free to continue the comments here and reopen if the problem persists after updating to the new version (to be released).

[M0uC0]

Status update on this issue with v4.3

With the new version it got to 24.54% at max speed and the panel rebooted by itself.

[20:13:33][D][nextion.upload.idf:238]: Uploaded 24.38 %, remaining 5667532 bytes, free heap: 3568 bytes [20:13:33][D][nextion.upload.idf:238]: Uploaded 24.43 %, remaining 5663436 bytes, free heap: 5288 bytes [20:13:34][D][nextion.upload.idf:238]: Uploaded 24.49 %, remaining 5659340 bytes, free heap: 4972 bytes [20:13:34][D][nextion.upload.idf:238]: Uploaded 24.54 %, remaining 5655244 bytes, free heap: 6624 bytes

The other panels i have with bluetooth proxy off all had free heap around 140000 bytes and no problems uploading the tft file.

Here i go.. disabling proxy.. uploading tft.. enabling proxy.. :)

[edwardtfn]

But do you have that code that stops the BLE when the upload TFT is called as shown here?

[M0uC0]

Not yet... i was trying to see if with the new version has is if it would upload :) I'll check if the script works later.

[edwardtfn]

I think we still have some space to save some RAM, but it is getting hard to find those bytes.

[M0uC0]

I've just added the script part to my yaml customization area Updated firmware and tried the TFT upload. It didn't managed to fully upload the tft.. it rebooted.

[22:26:17][D][addon_upload_tft.script.nextion_upload:129]: Starting TFT upload... [22:26:17][D][nextion.upload.idf:275]: Nextion TFT upload requested [22:26:17][D][nextion.upload.idf:276]: Exit reparse: YES [22:26:17][D][nextion.upload.idf:277]: URL: http://192.168.99.10:8123/local/nspanel_eu.tft [22:26:17][D][nextion.upload.idf:298]: Baud rate: 9600 [22:26:17][D][nextion.upload.idf:301]: Exiting Nextion reparse mode [22:26:17][D][nextion.upload.idf:411]: TFT file size: 7494348 bytes [22:26:17][D][nextion.upload.idf:428]: Uploading Nextion [22:26:22][D][nextion.upload.idf:468]: Upgrade response is [] - 0 byte(s) [22:26:22][E][nextion.upload.idf:474]: Preparation for TFT upload failed 0 "" [22:26:22][D][nextion.upload.idf:475]: Close HTTP connection [22:26:22][D][nextion.upload.idf:537]: Nextion TFT upload finished: Preparation for TFT upload failed [22:26:22][E][nextion.upload.idf:557]: Nextion TFT upload failed [22:26:22][D][addon_upload_tft.script.nextion_upload:131]: TFT upload: Preparation for TFT upload failed

Am i doing something wrong?

[plasterer41]

Did you reflash the panel with cable to install esp-idf. If you did, it will likely reboot a couple of times when you upload the tft file to the panel

[M0uC0]

it was already on esp-idf.. i have changed to esp-idf sometime ago.. and flashed via cable when i did it... has it is recommended..

the problem was available memory with bluetooth proxy on. And it is solved disabling bluetooth proxy to upload the tft file.

the problem now with the script to disable proxy while uploading is:

[22:26:22][E][nextion.upload.idf:474]: Preparation for TFT upload failed 0 ""

[edwardtfn]

It didn't managed to fully upload the tft.. it rebooted.

1. Could you please share a bit more of log?
2. Can you see how was the Free heap when you clicked the upload button?
3. Have you just tried again?

[M0uC0]

I didn't pasted the log because it's the same behaviour.. Around 20000 when starts.. and going down till around 5000 when it reboots.. It seems like nothing changed except the error I posted.

[edwardtfn]

Looks like the new ESPHome will be even worse:

v2024.2.2:

RAM:   [=         ]  10.2% (used 33276 bytes from 327680 bytes)
Flash: [======    ]  61.6% (used 1130105 bytes from 1835008 bytes)

v2024.3.0b1:

RAM:   [=         ]  10.3% (used 33596 bytes from 327680 bytes)
Flash: [======    ]  64.1% (used 1175629 bytes from 1835008 bytes)

[M0uC0]

Let's forget this problem... just disable bluetooth proxy.. upload.. enable again. As long has bluetooth proxy doesn't make the panel unstable.. :)

[edwardtfn]

But this is my main concern. You mentioned 20000 before upload, which is quite a few. If we keep loosing 320 on each ESPHome update it won't take long until this stops working.

[M0uC0]

Yes.. 20000 with bluetooth proxy.. and 140000 without.

When uploading without bluetooth proxy the heap is very stable... and always around 140000 during all the process. The problem with bluetooth is that it starts with 20000 but starts to decrease soon... and when it get bellow 5000 starts to slow down upload and finaly the panel reboots.

Am i missing something concerning the script !extend on upload_tft ?

[edwardtfn]

Am i missing something concerning the script !extend on upload_tft ?

That was a try to stop BLE before starting the upload, but looks like that was not effective. I will try to investigate this further.

[edwardtfn]

And it is solved disabling bluetooth proxy to upload the tft file.

I would recommend you also disable Upload TFT when BT proxy is enabled.

You can replace the packages like this:

substitutions:
  # Settings - Editable values
  device_name: "nsp-quarto" 
  wifi_ssid: !secret wifi_ssid
  wifi_password: !secret wifi_password
  nextion_update_url: "http://192.168.99.10:8123/local/nspanel_eu.tft"  # Optional for `esp-idf` framework
  # Add-on configuration (if needed)
  # heater_relay: "1"  # Possible values: "1" or "2"

# Customization area
##### My customization - Start #####

wifi:
  networks:
    - id: !extend wifi_default
    - ssid: !secret wifi_ssid2
      password: !secret wifi_password
  power_save_mode: LIGHT
# Set Wi-Fi power save mode to "LIGHT" as required for Bluetooth on ESP32

# Encrypt the communication between ESPHome and Home Assistant
api:
  encryption:
    key: !secret api_key
  reboot_timeout: 60min

# Use my global OTA password
ota:
  password: !secret ota_key

# Enable Bluetooth proxy
bluetooth_proxy:

##### My customization - End #####

# Core and optional configurations
packages:
  remote_package:
    url: https://github.com/Blackymas/NSPanel_HA_Blueprint
    ref: main
    files:
      # - nspanel_esphome.yaml                          # Basic package (core + tft upload)
      - esphome/nspanel_esphome_core.yaml               # Core package
      # - esphome/nspanel_esphome_addon_upload_tft.yaml # Upload TFT package (disable when BT Proxy is enabled)
      # Optional advanced and add-on configurations
      # - advanced/esphome/nspanel_esphome_advanced.yaml
      # - nspanel_esphome_addon_climate_cool.yaml
      # - nspanel_esphome_addon_climate_heat.yaml
      # - nspanel_esphome_addon_climate_dual.yaml
    refresh: 300s

esp32:
  framework:
    type: esp-idf
packages:
  core_package: !include 
  upload_tft_package: !include 
...

This will remove all the Upload TFT engine and free-up quite a lot of memory, which will probably give you a more stable system.

[M0uC0]

I'll try it out. Thanks

[M0uC0]

Upgraded to ESPHome to v4.3.1 with bluetooth proxy on. Tried to update TFT like that.

Started with: [15:21:46][D][nextion.upload.idf:285]: File size: 7497824 bytes [15:21:46][D][nextion.upload.idf:286]: Free heap: 30776 [15:21:46][D][nextion.upload.idf:026]: Range start: 0 [15:21:51][D][nextion.upload.idf:091]: Uploaded 0.05 %, remaining 7493728 bytes, free heap: 27240 bytes [15:21:51][D][nextion.upload.idf:095]: recv_string [08.00.00.62.00 (5)] [15:21:51][I][nextion.upload.idf:101]: Nextion reported new range 6422528 [15:21:51][D][nextion.upload.idf:026]: Range start: 6422528 [15:21:51][D][esp-idf:000]: I (82860) HTTP_CLIENT: Body received in fetch header state, 0x3fff8b9f, 89 [15:21:51][D][nextion.upload.idf:091]: Uploaded 85.71 %, remaining 1071200 bytes, free heap: 20452 bytes [15:21:51][D][nextion.upload.idf:091]: Uploaded 85.77 %, remaining 1067104 bytes, free heap: 22148 bytes

Finished with: [15:22:13][D][nextion.upload.idf:091]: Uploaded 99.70 %, remaining 22624 bytes, free heap: 10436 bytes [15:22:13][D][nextion.upload.idf:091]: Uploaded 99.75 %, remaining 18528 bytes, free heap: 8744 bytes [15:22:13][D][nextion.upload.idf:091]: Uploaded 99.81 %, remaining 14432 bytes, free heap: 10436 bytes [15:22:13][D][nextion.upload.idf:091]: Uploaded 99.86 %, remaining 10336 bytes, free heap: 7036 bytes [15:22:13][D][nextion.upload.idf:091]: Uploaded 99.92 %, remaining 6240 bytes, free heap: 8736 bytes [15:22:13][D][nextion.upload.idf:091]: Uploaded 99.97 %, remaining 2144 bytes, free heap: 12516 bytes [15:22:13][D][nextion.upload.idf:091]: Uploaded 100.00 %, remaining 0 bytes, free heap: 17480 bytes [15:22:13][D][nextion.upload.idf:307]: Successfully uploaded TFT to Nextion! [15:22:13][D][nextion.upload.idf:309]: Close HTTP connection [15:22:13][D][nextion.upload.idf:317]: Nextion TFT upload finished: Upload successful [15:22:13][D][nextion.upload.idf:328]: Restarting ESPHome

Managed to update tft without problems.. :)

[edwardtfn]

7kb of free heap is quite low. :( I will have to look on what is causing the memory issue. It could be something else running is parallel with the upload.

[M0uC0]

yes, it is low... and only managed to finish because it only needed to upload 15% of the file.

concerning the: packages: core_package: !include upload_tft_package: !include

I haven't tried it out.. i only have one panel with bluetooth proxy on.. and i don't have complaints regarding stability..

[edwardtfn]

I ran some test uploading TFT in an almost clean system (only basic package, with TFT upload, but no BLE) and transferred files multiple times (alternating between US and US Land to force it starting from 0), always at 115kbps. I see the free memory going a bit down in the first 30%, but then it gets quite stable. It fluctuates, which is expected, but nothing point to a leak.

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

  | Run 1 | Run 2 | Run 3 | Run 4 -- | -- | -- | -- | -- Before: | 200728 | 200692 | 200576 | 199784 Min: | 98528 | 99624 | 100368 | 98256 Max: | 110444 | 111068 | 111732 | 110404 Diff: | 11916 | 11444 | 11364 | 12148 Avg: | 104674 | 104329 | 105123 | 104520 Median: | 105412 | 104980 | 105744 | 105128 Sdev: | 2005 | 2151 | 2122 | 2066

Still the difference from the maximum free heap to minimum free heap is around 12kb, and I think the memory available when BLE is active is not giving space for this fluctuation. I can see some points for improvement on the ESPHome Nextion component, which could give us a few more bytes, so I will try to find some time to work at that.

But it's important to understand that BLE is leaving too few memory for other things, so this will probably postpone this issue, but not eliminating it. A proper solution should be freeing-up the BLE memory when a TFT upload starts, and looks like what we suggested in our customizations isn't getting that so, for now, I keep my recommendation to not have both packages (BLE and Upload TFT) installed at the same time, which I've mentioned on https://github.com/Blackymas/NSPanel_HA_Blueprint/issues/1815#issuecomment-2004597441.

[edwardtfn]

I'm looking for someone brave enough to run some tests with PSRAM. This is not solving the flash limitation related to using BLE, but may help with the Heap issue. We are having a discussion around this on #1983. I started testing it, but as I don't really use BLE, I don't have good parameters, so I'm looking for someone else facing memory issues to try this, but please be prepared to get a black screen and have to flash you panel via serial if something goes wrong (it haven't happened with me).

If you wanna try it, just add this as a customization to your panel's yaml:

esp32:
  framework:
    sdkconfig_options:
      CONFIG_D0WD_PSRAM_CLK_IO: "5"
      CONFIG_D0WD_PSRAM_CS_IO: "18"

psram:

[MichaelHeimann]

I'm looking for someone brave enough to run some tests with PSRAM. This is not solving the flash limitation related to using BLE, but may help with the Heap issue. We are having a discussion around this on #1983. I started testing it, but as I don't really use BLE, I don't have good parameters, so I'm looking for someone else facing memory issues to try this, but please be prepared to get a black screen and have to flash you panel via serial if something goes wrong (it haven't happened with me).

If you wanna try it, just add this as a customization to your panel's yaml:

esp32:
  framework:
    sdkconfig_options:
      CONFIG_D0WD_PSRAM_CLK_IO: "5"
      CONFIG_D0WD_PSRAM_CS_IO: "18"

psram:

I saw that discussion and tried the config change but for some reason my panel doesn't want to boot with that. It's stuck in the blue loading screen. Maybe because of flash being too: full

Linking .pioenvs/nsbuero/firmware.elf
RAM:   [==        ]  18.9% (used 61900 bytes from 327680 bytes)
Flash: [==========]  99.2% (used 1820721 bytes from 1835008 bytes)

I'll try with removing some stuff ....

[MichaelHeimann]

Jep - without the ble stuff, it loads and psram is there:

[edwardtfn]

Ok, but if that conflicts with BLE than it isn't really helping with the initial issue reported here. :(

[MichaelHeimann]

Yes, sadly.

We have a memory issue, find a way to use the additional already existing hardware RAM and then we run into a storage issue. bummer.

I did try to remove other stuff (debug, ipv6, heating addon...) and only leave the skeleton, ble and psram - it gets down to 98.9% flash and nearly boots (progress bar moves about a cm but then stalls. Adding all that in and just removing psram lands at 94% (!)

So nope. Is there any way to compile more ... space cautios? I read about psram bugs and fixes needing flash...

[edwardtfn]

I've tried with only the basic package and I can see the PSRAM component itself takes some memory, both RAM and Flash, and this is not neglectable:

v4.3.2d1 without PSRAM:

RAM:   [=         ]  10.7% (used 34948 bytes from 327680 bytes)
Flash: [======    ]  63.3% (used 1160921 bytes from 1835008 bytes)

v4.3.2d1 with PSRAM:

RAM:   [=         ]  11.1% (used 36532 bytes from 327680 bytes)
Flash: [=======   ]  67.3% (used 1235125 bytes from 1835008 bytes)

[MichaelHeimann]

compiling with CONFIG_ESP32_REV_MIN_3 reduces it again. But it won't boot still...

weird. are we sure, it's a space/flash issue? I mean 99% < 100% AND:

with psram and ble and CONFIG_ESP32_REV_MIN_3 but without heating addon it seems like space is there, but not booting to the end...

RAM:   [==        ]  18.4% (used 60284 bytes from 327680 bytes)
Flash: [========= ]  92.5% (used 1697933 bytes from 1835008 bytes)

just removing psram:

RAM:   [==        ]  18.3% (used 60124 bytes from 327680 bytes)
Flash: [========= ]  92.1% (used 1689925 bytes from 1835008 bytes)

and it boots.

[edwardtfn]

That memory shown when compiling is the estimated memory needed for the firmware to start, but after starting, more memory will be taken.

[edwardtfn]

But definitely could be something else crashing...

[MichaelHeimann]

Is there any reason not to use CONFIG_ESP32_REV_MIN_3 per default since it saves flash and all nspanels have a v3 ESP32 or am I not seeing someting?

It seems like it's also saving some 3kb heap...

Nope, that was just me forgetting to activate the climate addon again - it saved about 0,7kb heap But still: only improvements and not downsides, right?

[edwardtfn]

Is there any reason not to use CONFIG_ESP32_REV_MIN_3 per default since it saves flash and all nspanels have a v3 ESP32 or am I not seeing someting?

I've no reason to not try that. Just never tried. How are you setting it up?

[MichaelHeimann]

Is there any reason not to use CONFIG_ESP32_REV_MIN_3 per default since it saves flash and all nspanels have a v3 ESP32 or am I not seeing someting?

I've no reason to not try that. Just never tried. How are you setting it up?

esp32:
  framework:
    type: esp-idf
    sdkconfig_options:
      CONFIG_D0WD_PSRAM_CLK_IO: "5"
      CONFIG_D0WD_PSRAM_CS_IO: "18"
      CONFIG_ESP32_REV_MIN_3: "y"

I thought this is the way from the psram discussion. Maybe I'm wrong. But it seems to be effective...

[edwardtfn]

Why not! 😃

v4.3.2d1 without PSRAM:

RAM:   [=         ]  10.7% (used 34948 bytes from 327680 bytes)
Flash: [======    ]  63.3% (used 1160921 bytes from 1835008 bytes)

v4.3.2d1 with PSRAM:

RAM:   [=         ]  11.1% (used 36532 bytes from 327680 bytes)
Flash: [=======   ]  67.3% (used 1235125 bytes from 1835008 bytes)

v4.3.2d1 with PSRAM and CONFIG_ESP32_REV_MIN_3:

RAM:   [=         ]  10.7% (used 35068 bytes from 327680 bytes)
Flash: [======    ]  63.5% (used 1165833 bytes from 1835008 bytes)

[MichaelHeimann]

OK, I've tried all combinations I could think of. I give up. ble + psram wont startup 100%

I get different results with every boot. Resets during startup "loading bar" and sometimes I see the homescreen with clock and temperature but without everything else. HA icon is red, wireless logs cannot be fetched (again, once I could actually get a sneak peak) but ping always works. I can upload another firmware wirelessly when I try hard enough.

I doubt it's flash or ram, since it actually uses less on both than my working image. and think it's some kind of load dependant. Why?

firmware with all but psram: (booting fine)

RAM:   [=         ]  14.1% (used 46100 bytes from 327680 bytes)
Flash: [========= ]  94.2% (used 1729181 bytes from 1835008 bytes)

firmware with nothing but psram and ble: (not making it)

RAM:   [=         ]  14.1% (used 46068 bytes from 327680 bytes)
Flash: [========= ]  92.7% (used 1700417 bytes from 1835008 bytes)

https://github.com/Blackymas/NSPanel_HA_Blueprint/assets/3189562/97048573-54e8-4002-9d9c-8cd3e030b1ac

[edwardtfn]

Looks like it is connecting to WiFi, but not to API. It isn't consitenty. Sometimes I don't see even ESPHome connecting to Nextion, so I'm not sure where the thing is.

Can you get ESPHome log in that state? I'm asking, but as it isn't connecting to API, probably log isn't working also.

I will investigate this further. But not now. 💤

[MichaelHeimann]

I was trying to get a log but could barely make a connection.: logs_nsbuero_logs.txt

To me this looks like something is destroying the performance so bad that wifi is not working reliably anymore. ICMP is continuous but anything that needs a TCP connection is barely even making a three-way-handshake.

Maybe I could get bettler logs via serial. But... well.... not now. 💤 ;)

[edwardtfn]

When I remove Upload TFT, it works fine with PSRAM:

RAM:   [==        ]  17.8% (used 58316 bytes from 327680 bytes)
Flash: [========  ]  81.5% (used 1496269 bytes from 1835008 bytes)

[MichaelHeimann]

Soo.... new esphome 2024.3.1 improved little. But I'm kinda stubborn.

I've been reading about these CONFIG_ "cheatcodes" (mostly here) and it works now.

What did I do:

esp32:
  framework:
    type: esp-idf
    sdkconfig_options:
      CONFIG_D0WD_PSRAM_CLK_IO: "5"
      CONFIG_D0WD_PSRAM_CS_IO: "18"
      CONFIG_ESP32_REV_MIN_3: "y"
      CONFIG_BT_ALLOCATION_FROM_SPIRAM_FIRST: "y"
      CONFIG_BT_BLE_DYNAMIC_ENV_MEMORY: "y"
      CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC: "y"
      CONFIG_MBEDTLS_SSL_VARIABLE_BUFFER_LENGTH: "y"
      CONFIG_MBEDTLS_DYNAMIC_BUFFER: "y"
      CONFIG_MBEDTLS_DYNAMIC_FREE_CONFIG_DATA: "y"
      CONFIG_MBEDTLS_DYNAMIC_FREE_CA_CERT: "y"
      CONFIG_LWIP_MAX_SOCKETS: "5"
      CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP: "y"

And now I'm from

RAM:   [==        ]  18.4% (used 60276 bytes from 327680 bytes)
Flash: [========= ]  93.9% (used 1722641 bytes from 1835008 bytes)

to

RAM:   [=         ]  13.9% (used 45612 bytes from 327680 bytes)
Flash: [========= ]  92.2% (used 1691853 bytes from 1835008 bytes)

I have enabled ble, ipv6, debugger, psram, climate_addon - all I need.

I've also freed quite some heap ram:

I'll longterm test now.

[MichaelHeimann]

All seems great. Only issue so far: the "free heap" log lines during tft upload seems to be psram+heap (?). it's to high for sure...

[15:11:36][D][sensor:094]: 'Heap Free': Sending state 76964.00000 B with 0 decimals of accuracy
[15:11:36][D][sensor:094]: 'Heap Max Block': Sending state 43008.00000 B with 0 decimals of accuracy
[15:11:36][D][sensor:094]: 'Loop Time': Sending state 41.00000 ms with 0 decimals of accuracy
[15:11:36][D][sensor:094]: 'PSRAM Free': Sending state 2010035.00000 B with 0 decimals of accuracy
[15:11:37][D][esp32_ble_tracker:266]: Starting scan...
[15:11:37][D][sensor:094]: 'Ivanas Handy Büro RSSI': Sending state nan dBm with 0 decimals of accuracy
[15:11:39][D][addon_upload_tft.script.nextion_upload:129]: Starting TFT upload...
[15:11:39][D][nextion.upload.idf:128]: Nextion TFT upload requested
[15:11:39][D][nextion.upload.idf:129]: Exit reparse: NO
[15:11:39][D][nextion.upload.idf:130]: URL: http://10.250.23.6:8123/local/nspanel_eu.tft
[15:11:39][D][nextion.upload.idf:151]: Baud rate: 921600
[15:11:39][D][nextion.upload.idf:207]: TFT file size: 7497824 bytes
[15:11:39][D][nextion.upload.idf:220]: Uploading Nextion
[15:11:40][D][nextion.upload.idf:260]: Upgrade response is [05] - 1 byte(s)
[15:11:40][D][nextion.upload.idf:283]: Uploading TFT to Nextion:
[15:11:40][D][nextion.upload.idf:284]:   URL: http://10.250.23.6:8123/local/nspanel_eu.tft
[15:11:40][D][nextion.upload.idf:285]:   File size: 7497824 bytes
[15:11:40][D][nextion.upload.idf:286]:   Free heap: 2057035
[15:11:40][D][nextion.upload.idf:026]: Range start: 0
[15:11:45][D][nextion.upload.idf:091]: Uploaded 0.05 %, remaining 7493728 bytes, free heap: 2058803 bytes
[15:11:45][D][nextion.upload.idf:095]: recv_string [08.00.00.62.00 (5)]
[15:11:45][I][nextion.upload.idf:101]: Nextion reported new range 6422528
[15:11:45][D][nextion.upload.idf:026]: Range start: 6422528
[15:11:45][D][esp-idf:000]: I (310762) HTTP_CLIENT: Body received in fetch header state, 0x3fff4d87, 89
[15:11:45][D][nextion.upload.idf:091]: Uploaded 85.71 %, remaining 1071200 bytes, free heap: 2051415 bytes
[15:11:45][D][nextion.upload.idf:091]: Uploaded 85.77 %, remaining 1067104 bytes, free heap: 2053107 bytes
[15:11:45][D][nextion.upload.idf:091]: Uploaded 85.82 %, remaining 1063008 bytes, free heap: 2051415 bytes
[15:11:45][D][nextion.upload.idf:091]: Uploaded 85.88 %, remaining 1058912 bytes, free heap: 2053115 bytes

[edwardtfn]

the "free heap" log lines during tft upload seems to be psram+heap (?). it's to high for sure...

That's right! As I understand, I'm the end, the psram should be made available to all the components almost as regular Heap. It is a bit slower, but other than this most of the code should take advantage of that additional memory just like regular Heap. The problem is that psram is not available from the beginning of the loading process, so some components may still need regular memory while booting up.

[edwardtfn]

I will enable that on my ~3 panels~ 2 panels with idf (the other one is Arduino still) also and let's see if it is all stable.

[edwardtfn]

So, I've enabled basic (core + upload TFT), climate dual and advanced packages. Everything looks fine so far, but I can see the Component XXXXX took a long time for an operation (YY ms). warnings way more frequently now, which kind of is expected, as those are using a memory which is a bit slower than previously. And quite lot additional code competing for resources.

I will play later with psram mode and speed...

[Bascht74]

Another approach to make the TFT Update more stable could be to release the resources blocked by the BT features.

There is a discussion about it here: https://esp32.com/viewtopic.php?f=13&t=3139&sid=947d3f9b37003954564eb21970818822&start=10

(freeing 50kb - 70 kb of heap). As you are rebooting the esp after the TFT Update this should be no problem.

esp_err_t esp_bt_controller_mem_release(esp_bt_mode_t mode)

and

esp_err_t esp_bt_mem_release(esp_bt_mode_t mode)

Could this help in any way?

[MichaelHeimann]

Yet another aproach is using NimBLE instead of Bluedroid as written here

I can say that the logs of my 9 nspanels with psram and ble enabled are pretty. Sometimes I get

[W][component:232]: Component adc.sensor took a long time for an operation (65 ms).
[W][component:233]: Components should block for at most 30 ms.

But then I can't say if it's more often than before. I am happy to finally have ble AND tft upload capability. I'll look into using the nimble stack for the tracker. https://github.com/vgijssel/setup/pull/127 seems to be a drop-in replacement and some say to save as much as 70k memory (!).

[Bascht74]

Is there any reason not to use CONFIG_ESP32_REV_MIN_3 per default since it saves flash and all nspanels have a v3 ESP32 or am I not seeing someting?

Yes, this is a good idea to do that from what I read. We have V3 chips so any stuff that is about the bugs of V1/2 isn't needet.