Replacing the main application fails

[spuke113]

Hello @hn, I'm trying to flash esphome image to the Solis S3 WiFi Data Logging Stick (3rd gen). I followed your instructions, but the very last step fails with just a generic error:

user@linux-pc:~/rtltool/libretiny-esphome$ python3 -m esphome upload solis-inv-esphome.yaml --device /dev/ttyUSB0
INFO Reading configuration solis-inv-esphome.yaml...
Processing solis-inv (board: generic-rtl8710bx-4mb-980k; framework: arduino; platform: libretiny)

Configuring upload protocol...
AVAILABLE: uart
CURRENT: upload_protocol = uart
Looking for upload port...
Using manually specified: /dev/ttyUSB0
Uploading .pioenvs/solis-inv/firmware.uf2
|-- Detected file type: UF2 - esphome 2023.5.0-dev
|-- Connecting to 'Realtek AmebaZ' on /dev/ttyUSB0 @ 1500000
|   |-- Success! Chip info: Realtek RTL87xxB
|-- Writing '.pioenvs/solis-inv/firmware.uf2'
|   |-- esphome 2023.5.0-dev @ 2023-04-29 22:50:42 -> generic-rtl8710bx-4mb-980k
|-- ValueError: Failed to write to 0x800B000
|   |-- File "/home/user/.local/lib/python3.10/site-packages/ltchiptool/soc/ambz/flash.py", line 147, in flash_write_raw
*** [upload] Error 1

The stick is in bricked state after flashing fails. I managed to restore the stock firmware by running (MCU is in UART boot mode):

user@linux-pc:~/rtltool$ python3 rtltool.py -p /dev/ttyUSB0 wf 0x8000000 solis-s3-firmware-1012f.bin 
Connecting...
Write Flash data 0x08000000 to 0x08800000 from file: solis-s3-firmware-1012f.bin ...
Traceback (most recent call last):
  File "/home/user/rtltool/rtltool.py", line 523, in <module>
    main(*sys.argv[1:])
  File "/home/user/rtltool/rtltool.py", line 388, in main
    for _ in gen:
TypeError: 'bool' object is not iterable

Even though I get this error, the stick seems to work fine with the stock firmware.

Any ideas what I might be doing wrong?

[hn]

Well, things are a little bit shaky at the moment. @kuba2k2 does a really great job with LibreTiny, but it's not easy to have stable support for those cloned chips like the EMW3080.

The good thing is: it's doable -- I've got a stable WiFi S3 stick running with HA for some weeks.

With rtltool3.py I also experienced some errors, I reverted back to using rtltool2.py.

Try to flash the app binary alone:

python2 ./rtltool2.py -p /dev/ttyUSB0 wf 0xb000 ./.esphome/build/solis-inv/.pioenvs/solis-inv/image_0x00B000.ota1.bin

If this works and LibreTiny comes up (watch boot log, e.g. screen /dev/ttyUSB0 115200) you might experience WiFi problems with LibreTiny 1.0.1. Then you have to downgrade LibreTiny and apply some patches. Please report back, I'll help you with that.

[spuke113]

Thanks for quick response! Rtltool python2 version worked flawlessly.

As you expected, now I'm experiencing exactly the same WiFi issue you have. Any downgrade tips for downgrading to older LibreTiny would be much appreciated :)

[kuba2k2]

@hn could you post the rtltool2 and rtltool3 versions that you're using? ltchiptool is based on rtltool (an old version that worked on Python 2 I think) which makes it odd that it doesn't work. I could compare the working tool to the current code and maybe fix ltchiptool.

[hn]

Install LibreTiny v0.13.0 (may you have to remove the current version before, I dont know):

platformio platform install -f https://github.com/kuba2k2/libretiny#v0.13.0

This will probably install LibreTiny under its old name LibreTuya and you will have to adjust paths and/or names. Please report back so others can benefit from your learnings.

Apply this diff to make the UART work:

--- ./.platformio/platforms/libretuya/arduino/realtek-ambz/cores/arduino/SerialClass.cpp    2023-03-22 15:50:18.620881017 +0100
+++ ./.platformio/platforms/libretuya/arduino/realtek-ambz/cores/arduino/SerialClass.cpp    2023-03-22 16:50:18.620881017 +0100
@@ -27,9 +27,10 @@
    data->uart->DLH_INTCR = 0;

    uint8_t c;
-   UART_CharGet(data->uart, &c);
-   if (c)
-       data->buf->store_char(c);
+   while (UART_Readable(data->uart)) {
+       UART_CharGet(data->uart, &c);
+       data->buf->store_char(c);
+   }

    data->uart->DLH_INTCR = intcr;
    return 0;

Compile and upload the binary to the WiFi stick as before. Subsequent updates can be carried out via OTA.

If you experience ModBus errors after some hours (e.g. no new values show up in HA), you may have to apply this "makes-no-sense-but-helps" fix as well: https://github.com/kuba2k2/libretiny/issues/91#issuecomment-1520805193

[spuke113]

I'm running into a lot of compiling errors. So far, I made the changes you suggested and git checked out latest commit from libretiny-esphome before name change (commit SHA 7d37eea0f731a55274659ca08662dd96386e38dd). I also changed libretiny->tuya in solis-inv-esphome.yaml. I'm currently trying to sort out this error:

Compiling .pioenvs/solis-inv/src/esphome/components/libretuya/preferences.cpp.o
src/esphome/components/libretuya/core.cpp: In function 'void esphome::arch_init()':
src/esphome/components/libretuya/core.cpp:22:3: error: 'lt_wdt_enable' was not declared in this scope
   22 |   lt_wdt_enable(10000L);
      |   ^~~~~~~~~~~~~
src/esphome/components/libretuya/core.cpp:24:3: error: 'lt_gpio_recover' was not declared in this scope
   24 |   lt_gpio_recover();
      |   ^~~~~~~~~~~~~~~
src/esphome/components/libretuya/core.cpp: In function 'void esphome::arch_restart()':
src/esphome/components/libretuya/core.cpp:29:3: error: 'lt_reboot' was not declared in this scope
   29 |   lt_reboot();
      |   ^~~~~~~~~
src/esphome/components/libretuya/core.cpp: In function 'void esphome::arch_feed_wdt()':
src/esphome/components/libretuya/core.cpp:33:38: error: 'lt_wdt_feed' was not declared in this scope
   33 | void IRAM_ATTR HOT arch_feed_wdt() { lt_wdt_feed(); }
      |                                      ^~~~~~~~~~~
src/esphome/components/libretuya/core.cpp: In function 'uint32_t esphome::arch_get_cpu_cycle_count()':
src/esphome/components/libretuya/core.cpp:34:46: error: 'lt_cpu_get_cycle_count' was not declared in this scope
   34 | uint32_t arch_get_cpu_cycle_count() { return lt_cpu_get_cycle_count(); }
      |                                              ^~~~~~~~~~~~~~~~~~~~~~
src/esphome/components/libretuya/core.cpp: In function 'uint32_t esphome::arch_get_cpu_freq_hz()':
src/esphome/components/libretuya/core.cpp:35:42: error: 'lt_cpu_get_freq' was not declared in this scope
   35 | uint32_t arch_get_cpu_freq_hz() { return lt_cpu_get_freq(); }
      |                                          ^~~~~~~~~~~~~~~
Compiling .pioenvs/solis-inv/src/esphome/components/logger/logger.cpp.o
*** [.pioenvs/solis-inv/src/esphome/components/libretuya/core.cpp.o] Error 1
In file included from src/esphome/components/sensor/sensor.h:3,
                 from src/esphome/components/libretuya/lt_component.h:9,
                 from src/esphome/components/libretuya/lt_component.cpp:1:
src/esphome/components/libretuya/lt_component.cpp: In member function 'virtual void esphome::libretuya::LTComponent::dump_config()':
src/esphome/components/libretuya/lt_component.cpp:14:39: error: 'lt_get_version' was not declared in this scope
   14 |   ESP_LOGCONFIG(TAG, "  Version: %s", lt_get_version());
      |                                       ^~~~~~~~~~~~~~
src/esphome/core/log.h:97:90: note: in definition of macro 'esph_log_config'
   97 |   esp_log_printf_(ESPHOME_LOG_LEVEL_CONFIG, tag, __LINE__, ESPHOME_LOG_FORMAT(format), ##__VA_ARGS__)
      |                                                                                          ^~~~~~~~~~~
src/esphome/components/libretuya/lt_component.cpp:14:3: note: in expansion of macro 'ESP_LOGCONFIG'
   14 |   ESP_LOGCONFIG(TAG, "  Version: %s", lt_get_version());
      |   ^~~~~~~~~~~~~
src/esphome/components/libretuya/lt_component.cpp:19:35: error: 'LT_BANNER_STR' was not declared in this scope; did you mean 'LT_BANNER'?
   19 |     this->version_->publish_state(LT_BANNER_STR);
      |                                   ^~~~~~~~~~~~~
      |                                   LT_BANNER
*** [.pioenvs/solis-inv/src/esphome/components/libretuya/lt_component.cpp.o] Error 1

I'm very new to all of this, but I'll report back as soon as I figure this out.

[kuba2k2]

You'd have to check out the latest commit before the refactor in esphome. The name change commit was a bit after the refactor commit.

[hn]

@spuke113 I used libretiny-esphome from mid-march https://github.com/kuba2k2/libretiny-esphome/commit/d076454f6f9db93169e3a5e090e8ab94830caf5e

[spuke113]

It works! At least for now. https://github.com/kuba2k2/libretiny-esphome/commit/d076454f6f9db93169e3a5e090e8ab94830caf5e commit compiled just fine. I did not uninstall current platformio libretiny - I just installed older version on top of the current one. I made the suggested change in SerialClass.cpp. No other changes were needed, except in solis-inv-esphome.yaml libretiny was renamed back to libretuya. I did not apply that modbus "fix", at least not yet since the stick has been working flawlessly for the past 8 hours.

I don't have UART-USB device connected to the stick at the moment, since it's kinda tricky to access the UART pins while the stick is attached to the inverter (without drilling holes to the case, warranty void bla bla...). Therefore I don't have access to debug logs to monitor status of the modbus, but I suppose the problem shows itself as generic instability in HA as well.

Thanks for useful tips! I'll report back in a couple of days.

[kuba2k2]

@hn we've fixed a malloc() issue just yesterday, which I haven't noticed - it is possible that it was the source of your WiFi problems. Could you check the new version when you get the chance?

[hn]

@spuke113 : Congratulations, you are registered user # 2 for wifi-stick-s3-open-source :-)

For connecting the UART (without voiding warranty -- a good joke if you re-flashed the firmware some minutes ago), I completely removed the PCB vom the case and just plugged in the (black) cap into the inverter. Then I pushed the PCB into the cap, it it sits there quite firmly, even without the case. Warning: do not confuse the orientation of the PCB!

@kuba2k2 : Great, just give me 1-2 days.

[hn]

@kuba2k2 : Just a quick test with 1.0.1+sha.86924d8: WiFi connect works, even WiFi Captive Portal works for the very first time with the EMW3080, hooray :) Did not test ModBus yet, but looks promising.

[kuba2k2]

Note to self: don't use linker aliases.

Let me know if it's stable in a few days.

[spuke113]

@hn The stick worked fine 3 days in a row. Yesterday updates stopped flowing to HA and sure shit modbus was throwing crc errors. I applied that Serial2.println("foo"); "fix" today morning, but only a couple of hours later crc errors were back again. Booo :(

I have a logic analyzer, I'll try to sniff the traffic between the stick and the inverter tomorrow. Deep rabbit hole, here we come!

[hn]

@spuke113 I think the ModBus traffic is ok on wire, for some reason it gets de-synced (all the data is there, but incorrectly shifted into the individual telegrams).

I've opened a new issue to have one place to track this.

[Belaial]

After finally solving my issues in this ticket https://github.com/hn/ginlong-solis/issues/9 I tonight tried to replace the main application, however it's not going great.... all works except the last command

I get the following error

============================================

[sassa@fedora libretiny-esphome]$ python3 -m esphome upload solis-esphome-emw3080.yaml --device /dev/ttyUSB0 INFO ESPHome 2023.7.0-dev INFO Reading configuration solis-esphome-emw3080.yaml...

Obsolete PIO Core v6.1.7 is used (previous was 6.1.9) Please remove multiple PIO Cores from a system: https://docs.platformio.org/en/latest/core/installation/troubleshooting.html

Processing solis-emw3080 (board: generic-rtl8710bx-4mb-980k; framework: arduino; platform: libretiny)

Configuring upload protocol... AVAILABLE: uart CURRENT: upload_protocol = uart Looking for upload port... Using manually specified: /dev/ttyUSB0 Uploading .pioenvs/solis-emw3080/firmware.uf2 |-- Detected file type: UF2 - esphome 2023.7.0-dev |-- Connecting to 'Realtek AmebaZ' on /dev/ttyUSB0 @ 1500000 |-- SerialException: [Errno 13] could not open port /dev/ttyUSB0: [Errno 13] Permission denied: '/dev/ttyUSB0' | |-- File "/home/sassa/.local/lib/python3.11/site-packages/serial/serialposix.py", line 325, in open *** [upload] Error 1 [FAILED] Took 0.92 seconds

============================================

If I try with sudo I get this

============================================

sudo python3 -m esphome upload solis-esphome-emw3080.yaml --device /dev/ttyUSB0 Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/sassa/Desktop/jonas/libretiny-esphome/esphome/main.py", line 10, in from esphome import const, writer, yaml_util File "/home/sassa/Desktop/jonas/libretiny-esphome/esphome/writer.py", line 7, in from esphome.config import iter_components File "/home/sassa/Desktop/jonas/libretiny-esphome/esphome/config.py", line 11, in import voluptuous as vol ModuleNotFoundError: No module named 'voluptuous'

============================================

I am sure the ttyUSB0 port is the correct one by checking dmesg I have tried to unplug and re-plug the serial adapter several times I have tried rebooting I have tried a different PC Just to make sure that the port is not used by something else

Whatever I try I get the same results.... I am starting to feel cursed as soon as I touch this project.

Any ideas what's going wrong?

EDIT

I tried this trick (I know it's a dirty one and could be solved with adding my user to the correct group but I was lazy)

sudo chmod -R 777 /dev/ttyUSB0 from this thread https://github.com/esp8266/source-code-examples/issues/26

That solved my previous error however I got a new one, same as @spuke113

|-- ValueError: Failed to write to 0x800B000 | |-- File "/home/user/.local/lib/python3.10/site-packages/ltchiptool/soc/ambz/flash.py", line 147, in flash_write_raw *** [upload] Error 1

So now I'm sitting here with a bricked stick as well..... This is really getting on my nerves.... time for a break...

[hn]

No worries, we'll get it going :)

Please try to solely flash the app alone (ota1 and ota2 image) using the Python2 version of rtltool:

python2 ./rtltool2.py -p /dev/ttyUSB0 wf 0xb000 ./.esphome/build/solis-emw3080/.pioenvs/solis-emw3080/image_0x00B000.ota1.bin
python2 ./rtltool2.py -p /dev/ttyUSB0 wf 0x100000 ./.esphome/build/solis-emw3080/.pioenvs/solis-emw3080/image_0x100000.ota2.bin

[Belaial]

Not to worried :) Just annoyed that I am not more of a "hacker" :smile: I am sure we will get this sorted sooner or later.

I checked that I could still talk to the stick after bricking it, with the python2 ./rtltool.py -p /dev/ttyUSB0 gf and I got the Flash Status value: 0x40 so I assume that means it's still alive.

I did a quick search regarding the files you mentioned, image_0x00B000.ota1.bin and image_0x100000.ota2.bin but I can't locate them.

Thinking the best thing I could do is start over fresh tomorrow with a cup of coffee and a good nights sleep, plan is to install a fresh Linux Mint 21.1 and go from there, this laptop I am using now has been a "battle field" many years for different projects so I have a bad feeling about it.

(also a big thank you @hn for helping out all this time, your patience is amazing!)

[hn]

If you have an uf2 file (your logUploading .pioenvs/solis-emw3080/firmware.uf2), the ota images have been created and there is no need to wipe your operating system. Try find . -name image_0x00B000.ota1.bin in the libretuya-esphome directory where the yaml file is placed.

[Belaial]

So, I did not wait for either coffee or a good nights sleep :smiley: I started fresh and really read every log during the installation! (that is also how I found the exact paths to my compiled files)

I did exactly what you told me to in https://github.com/hn/ginlong-solis/issues/2#issuecomment-1625731577 and what do you know!

F*** YES!!!!!!!!!!!!!!! FINALLY!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!