search

emsesp / EMS-ESP32

ESP32 firmware to read and control EMS and Heatronic compatible equipment such as boilers, thermostats, solar modules, and heat pumps
https://emsesp.github.io/docs
GNU Lesser General Public License v3.0
579 stars 101 forks source link

Supporting latest Arduino Core v3 - issues with fixed Network IP address, settings lost after upgrade etc #1804

Open tp1de opened 3 months ago

tp1de commented 3 months ago

Being in my holiday home with netfit thermostat moduline 2050 (ems 2.0) I recognized that the thermostat_hc1_mode does not work anymore (neither dashboard nor mqtt). No changes possible - no error reported.

EMS-ESP version 3.7.0-dev.10 works ok, but EMS-ESP version 3.7.0-dev.12 and 13 are not working. On my EMS+ system with RC310 versions EMS-ESP version 3.7.0-dev.12 and 13 are working fine.

What has been changed?

tp1de commented 3 months ago

.... and while installing 3.7.0-dev.13 both systems do not restart anymore. My home system with E32 needed a hard reboot and was working again. My S3 system within my holiday home lost all config data and wifi started with access point. This is not good.

MichaelDvP commented 3 months ago

For mode see #1800, should be fixed.

The new dev is compiled with arduino 3.0.1. Maybe the newer littleFS version reformats on an old system. I've seen this a long time ago when frst trying the arduino 3.xx. But since that i can change between 2.xx and 3.xx versions without issue.

proddy commented 3 months ago

This is indeed not good. We'll need to test the migration and see what is causing the FS issue. If it's a problem for others we should perhaps better remove the firmware bin files from the dev repo

MichaelDvP commented 3 months ago

I don't think it'a a general issue, Harvay in #1800 and others don't have FS cleared on update. I think there was one specific version in the past and systems USB-flashed with this version are not recognized y the arduino-3-LittleFS. I have seen it a very few time, but can not reproduce and find the specific version.

tp1de commented 3 months ago

For mode see #1800, should be fixed.

The fix is still not within latest dev13 .... or?

mattreim commented 3 months ago

Version 3.7.0-dev.13(3) from yesterday has been running for me for 24 hours without any problems.

tp1de commented 3 months ago

I tested again: dev.13 was not working and after uploading bin-file all settings where lost on my S3 gateway again. So I flashed the gateway from my laptop , loaded the settings and afterwards it was working again. The thermostat.hc1.mode was changeable. So far so good.

But I recognized that the network settings in respect to fixed ip address and hostname where loaded but not used.
(dhcp instead). Fixed ip was only used after some retries. The defined hostname is not populated on the network.

mattfro commented 3 months ago

I have esp32 s3, after going from dev12 to dev14..I lost wifi, esp-ems went into AP mode. In AP mode I tried to use wifi scan to search for networks, it did not find anything. Going back to dev12 it worked again. In the end I even tried to manually install dev14 with ESP flasher and in that point I think all my settings was lost, but still wifi scan did not work. Stupid me, did not even think of setting wifi manually before this :) Because that seems to work in the end. Now running dev14 after setting back all settings like mqtt and wifi. It seems to work as it should

proddy commented 3 months ago

Thanks @mattfro - there are definitely something compatibility issues with the LittleFS file system that stores the EMS-ESP settings between Espressif Core (v2 on dev 12 and before) and the newer builds on Core v3.0.1 (dev 13 + 14). When I'm back home I'll need to run some tests, flashing with dev12 on an S3 and trying the upgrade. It may also be something related to the partitions or how the BBQKees gateways are originally flashed - we'll see. For now I would stick to dev12.

MichaelDvP commented 3 months ago

I don't know why s3 compiles with arduino core 3.xx. There seems to be a bug in platformio. We use platform = espressif32 which should use the latest stable from https://registry.platformio.org/platforms/platformio/espressif32, which is 6.7.0 with arduino core 2.0.16. Also https://github.com/platformio/platform-espressif32/releases says latest release is 6.7.0.
Using the development branch platform=https://github.com/platformio/platform-espressif32.git compiles with arduino 2.0.17.

In https://github.com/MichaelDvP/EMS-ESP32/releases i have the actual dev14 compiled with arduino core 2 (s3: espessif32@6.7.0, esp32: tasmota@2024-05-00)

With the arduino 3.0.1 i don't see issues with littleFS and fixed ip, but i can confirm that network scanner is not working with arduino 3.0.1 and 3.0.0 builds.

proddy commented 3 months ago

Maybe just explicitly add the 6.7.0 to the platform as we did last time.

For WiFi maybe it's related to the event names https://randomnerdtutorials.com/esp32-migrating-version-2-to-3-arduino/#wifi-event-name-changes

I'm not home so can't test at the moment

MichaelDvP commented 3 months ago

Maybe just explicitly add the 6.7.0 to the platform as we did last time.

That's exactly what i have done for the test: see https://github.com/emsesp/EMS-ESP32/commit/b1750265aa9a4bb1365da973b447c5a6f45346d9

BTW: the S3 arduino3.0.1builds are also very small, like the tasmota-arduino without TSL, only 1.6M bin file But with:

platform_packages=
  platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#3.0.1
  platformio/framework-arduinoespressif32-libs @ https://github.com/espressif/esp32-arduino-libs.git#idf-release/v5.1

i get 2.1M bin file. (But also network scan with empty result)

The event names are changed since arduino 2.0.12, in 3.xx the old ones are removed and throw error when using.

proddy commented 3 months ago

I made a dev14 that uses the previous 2.0.17 Arduino espressif32 library

tp1de commented 3 months ago

It is dev 15 and seems to work as before. WiFi scan is ok.

mattfro commented 3 months ago

Same here... everything seems to work now.

proddy commented 3 months ago

thanks @tp1de and @mattfro for quickly testing. I think we'll keep to the old Ardunio library until we see the benefits of moving to v3 of the core and have fully tested the Network and Filesystem changes.

Interesting article: https://www.cnx-software.com/2024/06/01/espressif-releases-arduino-esp32-core-3-0-0-but-platformio-support-in-doubt/

proddy commented 2 months ago

thanks @tp1de and @mattfro for quickly testing. I think we'll keep to the old Ardunio library until we see the benefits of moving to v3 of the core and have fully tested the Network and Filesystem changes.

Interesting article: https://www.cnx-software.com/2024/06/01/espressif-releases-arduino-esp32-core-3-0-0-but-platformio-support-in-doubt/

Related to espressif/platformio's little argument and further development, some of the devs have started creating their own pio fork: https://github.com/pioarduino/platform-espressif32

tp1de commented 2 months ago

@proddy @MichaelDvP while updating to latest dev.26 I lost all settings again. After re-setting I recognized that within ioBroker adapter all axios get request from devices failed (e.g. .../api/boiler etc.)

What has been changed?

proddy commented 2 months ago

Best to comment in https://github.com/emsesp/EMS-ESP32/issues/1806 - this issue is about core v3 and we're still running on v2

MichaelDvP commented 2 months ago

Not in #1806. This is the new message: https://github.com/emsesp/EMS-ESP32/blob/55476a7828e59ac9ad9e9c18fb9d02e95fb7043d/src/emsesp.cpp#L758-L762 which is shown in api/boiler/values etc. in first place. but is no entity.

proddy commented 2 months ago

oh, yes, I'll fix that

MichaelDvP commented 2 months ago

I've made a PR to clear output if there is a command function. @tp1de Were all settings cleared? have you changed between espressif-arduino and tasmota-arduino (e.g. 16M/4M version)? Is it reproducible?

proddy commented 2 months ago

oh, yes, I'll fix that

fixed now with https://github.com/emsesp/EMS-ESP32/pull/1879. Sorry about that

proddy commented 2 months ago

I see 6.8.0 is out (still on 2.0.17) so will try that https://github.com/platformio/platform-espressif32/releases/tag/v6.8.0

proddy commented 1 month ago

I'd like to try out https://github.com/pioarduino/platform-espressif32 too

MichaelDvP commented 1 month ago

It's updated very often, don't know if these are bugfixes or enhancements. I've tested a few times on ESP32S3 with PSRAM. Works. Comparing arduino 2.0.17 (espressif develop) to 3.04 (pioarduino develop) filesize 2.3 MB 2.6 MB free ram 201 198 max alloc 187 183 free caps 8343 8337 free main stack 4720 3852 free TCP stack 936 88

Free TCP stack is very low (setting to 6144), maybe increase to 7 or 8k.

Edit: it was set to 5k for arduino 3, so stack tcp stack usage is same for both versions, setting of 6k is ok.

No issues when switching between the builds, littleFS read all the configs. I only get littleFS issues when switching between tasmota and espressif builds.

Edit: Network scan always returns no results on arduino 3.

proddy commented 1 month ago

There's quite a lively discord channel with all the famous people (Jason2866, me-no-dev, E.M. etc) on it discussing the differences. I sent you an invite, but I know you dislike Discord ;-) Jason was talking about the differences in the Tasmota build compared to this (apparently has MQTT, Rainmaker, WifiProv and BLE all removed)

proddy commented 1 month ago

Edit: Network scan always returns no results on arduino 3.

would be interesting to see if this works on Arduno 3.0.4 https://raw.githubusercontent.com/RuiSantosdotme/Random-Nerd-Tutorials/master/Projects/ESP32/ESP32_WiFiScan.ino

proddy commented 4 weeks ago

Tried with the latest pioardunio (platform=https://github.com/pioarduino/platform-espressif32/releases/tag/53.03.10+rc1) which is Arduino v3.1.0/IDF v5.3.3 and comparing to Arduino v2.0.17/IDF v4.4.7 and both freemem and max alloc went down by 20% which isn't good.

edit Back with 3.0.4/IDF 5.1 things are better, only 5KB less memory. But WiFi scan is still broken.