Shelly initial OTA flash

copyrights commented 5 months ago

Describe the problem you have/What new integration you would like

It would be a nice feature to flash a factory new Shelly device (e.g. Shelly 1 mini gen. 3) over the air instead of opening and soldering. It should be possible as Tasmota uses mgos32-to-tasmota32 for this Please describe your use case for this integration and alternatives you've tried:

Use case: Flash ESPHome to shelly devices without soldering.

alternative: Open devices, solder wires to pbc, flash via serial.

Additional context

coolstuff99 commented 3 months ago

Is it possible / is there any progress on this topic? Thank you,

0xFEEDC0DE64 commented 1 month ago

Is it possible to first do the OTA update to tasmota and then do a second ota update to esphome?

copyrights commented 1 month ago

@0xFEEDC0DE64 yes, with the help of @angelnu 's work in #5535 .

After cloning of https://github.com/angelnu/esphome-1/tree/extend_ota and the prepartion of a pipenv for that I do the following steps.

Flash Tasmoto mgos32-to-tasmota32-Mini1G3.zip
Connect Tasmoto to IoT Wifi
Configure the device using Tasmota Auto configuration. (Configuration

Auto-configuration > Select new auto-configuration) Select your device and hit enter. This replaces the locked bootloader (without this step device gets bricked with the next boot).
Under consoles, open the Partition Wizard and start the Safeboot Conversion by hitting the button "Start Migration". The conversion will update to the latest Tasmota version too.
Use Partition Wizard to increase the Filesystem size to its maximum. This removes all obsolete files too.
cd ~/projects/esphome-1
bin/pipenv shell
esphome compile dlt.yaml
flash .esphome/build/s1mg3-dboden-l-tuer/.pioenvs/s1mg3-dboden-l-tuer/firmware.bin
sudo vim /etc/hosts
- 192.168.112.103 s1mg3-dboden-l-tuer.local
esphome -v upload-factory-ota dlt.yaml
then continue in esphome (prod instance)

Works fine, but are obviously a few extra steps. On the other hand: no soldering or open the box required.

0xFEEDC0DE64 commented 4 weeks ago

I am failing at step bin/pipenv shell because this esphome-1 repo doesnt even have a bin folder?

0xFEEDC0DE64 commented 4 weeks ago

update, fixed issues with pipenv

Can I see your esphome yaml file? Do I need to speficy anything additionally for bootloader partition-table and so on?

0xFEEDC0DE64 commented 4 weeks ago

$ python -m esphome compile ~/esphomes/klo-deckenlicht.yaml 
INFO ESPHome 2024.1.0-dev
INFO Reading configuration /home/feedc0de/esphomes/klo-deckenlicht.yaml...
INFO Detected timezone 'Europe/Vienna'
INFO Generating C++ source...
INFO Core config or version changed, cleaning build files...
INFO Deleting /home/feedc0de/esphomes/.esphome/build/klo-deckenlicht/.pioenvs
INFO Deleting /home/feedc0de/esphomes/.esphome/build/klo-deckenlicht/.piolibdeps
INFO Compiling app...
**********************************************************************************************************************************************************************************
Obsolete PIO Core v6.1.11 is used (previous was 6.1.15)
Please remove multiple PIO Cores from a system:
https://docs.platformio.org/en/latest/core/installation/troubleshooting.html
**********************************************************************************************************************************************************************************
Processing klo-deckenlicht (board: esp32-c3-devkitm-1; framework: espidf; platform: platformio/espressif32@5.4.0)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Tool Manager: Installing platformio/framework-espidf @ ~3.40406.0
INFO Installing platformio/framework-espidf @ ~3.40406.0
Downloading  [####################################]  100%          
Unpacking  [####################################]  100%          
Tool Manager: framework-espidf@3.40406.240122 has been installed!
INFO framework-espidf@3.40406.240122 has been installed!
Tool Manager: Installing platformio/tool-scons @ ~4.40502.0
INFO Installing platformio/tool-scons @ ~4.40502.0
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Tool Manager: tool-scons@4.40502.0 has been installed!
INFO tool-scons@4.40502.0 has been installed!
Tool Manager: Removing tool-scons @ 4.40700.0
INFO Removing tool-scons @ 4.40700.0
Tool Manager: tool-scons@4.40700.0 has been removed!
INFO tool-scons@4.40700.0 has been removed!
Library Manager: Installing esphome/noise-c @ 0.1.4
INFO Installing esphome/noise-c @ 0.1.4
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Library Manager: noise-c@0.1.4 has been installed!
INFO noise-c@0.1.4 has been installed!
Library Manager: Resolving dependencies...
INFO Resolving dependencies...
Library Manager: Installing esphome/libsodium @ 1.10018.1
INFO Installing esphome/libsodium @ 1.10018.1
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Library Manager: libsodium@1.10018.1 has been installed!
INFO libsodium@1.10018.1 has been installed!
HARDWARE: ESP32C3 160MHz, 320KB RAM, 4MB Flash
 - framework-espidf @ 3.40406.240122 (4.4.6) 
 - tool-cmake @ 3.16.4 
 - tool-ninja @ 1.7.1 
 - toolchain-esp32ulp @ 2.35.0-20220830 
 - toolchain-riscv32-esp @ 8.4.0+2021r2-patch5
ModuleNotFoundError: No module named 'pkg_resources':
  File "/home/feedc0de/.local/share/virtualenvs/esphome-1-WzRomffq/lib/python3.12/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/feedc0de/.platformio/platforms/espressif32@5.4.0/builder/main.py", line 312:
    target_elf = env.BuildProgram()
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
    return self.method(*nargs, **kwargs)
  File "/home/feedc0de/.local/share/virtualenvs/esphome-1-WzRomffq/lib/python3.12/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
    return self.method(*nargs, **kwargs)
  File "/home/feedc0de/.local/share/virtualenvs/esphome-1-WzRomffq/lib/python3.12/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
    return self.method(*nargs, **kwargs)
  File "/home/feedc0de/.local/share/virtualenvs/esphome-1-WzRomffq/lib/python3.12/site-packages/platformio/builder/tools/piobuild.py", line 342:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 662:
    return method(*args, **kw)
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/feedc0de/.platformio/platforms/espressif32@5.4.0/builder/frameworks/espidf.py", line 29:
    import pkg_resources
========================================================================== [FAILED] Took 19.57 seconds ==========================================================================

copyrights commented 4 weeks ago

update, fixed issues with pipenv

Can I see your esphome yaml file? Do I need to speficy anything additionally for bootloader partition-table and so on?

esphome:
  name: s1mg3-dboden-r-tuer
  platformio_options:
    board_build.flash_mode: dio
esp32:
  board: esp32-c3-devkitm-1
  framework:
    type: esp-idf
    platform_version: 6.4.0
    version: 5.1.1
  variant: esp32c3
wifi:
  networks:
  - ssid: IoT
    password: "xxxxxxxxxxxxxxxxxxxxxxxx"
ota:
  password: "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  unprotected_writes: True # This is mandatory if you want to flash the partition table or bootloader!
logger:
  hardware_uart: USB_SERIAL_JTAG
web_server:
button:
- platform: restart
  name: Restart

0xFEEDC0DE64 commented 4 weeks ago

the esphome fork has successfully compiled :)

now upload the firmware through tasmota webinterface or through esphome directly?

copyrights commented 4 weeks ago

$ python -m esphome compile ~/esphomes/klo-deckenlicht.yaml 
INFO ESPHome 2024.1.0-dev
INFO Reading configuration /home/feedc0de/esphomes/klo-deckenlicht.yaml...
INFO Detected timezone 'Europe/Vienna'
INFO Generating C++ source...
INFO Core config or version changed, cleaning build files...
INFO Deleting /home/feedc0de/esphomes/.esphome/build/klo-deckenlicht/.pioenvs
INFO Deleting /home/feedc0de/esphomes/.esphome/build/klo-deckenlicht/.piolibdeps
INFO Compiling app...
**********************************************************************************************************************************************************************************
Obsolete PIO Core v6.1.11 is used (previous was 6.1.15)
Please remove multiple PIO Cores from a system:
https://docs.platformio.org/en/latest/core/installation/troubleshooting.html
**********************************************************************************************************************************************************************************
Processing klo-deckenlicht (board: esp32-c3-devkitm-1; framework: espidf; platform: platformio/espressif32@5.4.0)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Tool Manager: Installing platformio/framework-espidf @ ~3.40406.0
INFO Installing platformio/framework-espidf @ ~3.40406.0
Downloading  [####################################]  100%          
Unpacking  [####################################]  100%          
Tool Manager: framework-espidf@3.40406.240122 has been installed!
INFO framework-espidf@3.40406.240122 has been installed!
Tool Manager: Installing platformio/tool-scons @ ~4.40502.0
INFO Installing platformio/tool-scons @ ~4.40502.0
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Tool Manager: tool-scons@4.40502.0 has been installed!
INFO tool-scons@4.40502.0 has been installed!
Tool Manager: Removing tool-scons @ 4.40700.0
INFO Removing tool-scons @ 4.40700.0
Tool Manager: tool-scons@4.40700.0 has been removed!
INFO tool-scons@4.40700.0 has been removed!
Library Manager: Installing esphome/noise-c @ 0.1.4
INFO Installing esphome/noise-c @ 0.1.4
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Library Manager: noise-c@0.1.4 has been installed!
INFO noise-c@0.1.4 has been installed!
Library Manager: Resolving dependencies...
INFO Resolving dependencies...
Library Manager: Installing esphome/libsodium @ 1.10018.1
INFO Installing esphome/libsodium @ 1.10018.1
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Library Manager: libsodium@1.10018.1 has been installed!
INFO libsodium@1.10018.1 has been installed!
HARDWARE: ESP32C3 160MHz, 320KB RAM, 4MB Flash
 - framework-espidf @ 3.40406.240122 (4.4.6) 
 - tool-cmake @ 3.16.4 
 - tool-ninja @ 1.7.1 
 - toolchain-esp32ulp @ 2.35.0-20220830 
 - toolchain-riscv32-esp @ 8.4.0+2021r2-patch5
ModuleNotFoundError: No module named 'pkg_resources':
  File "/home/feedc0de/.local/share/virtualenvs/esphome-1-WzRomffq/lib/python3.12/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/feedc0de/.platformio/platforms/espressif32@5.4.0/builder/main.py", line 312:
    target_elf = env.BuildProgram()
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
    return self.method(*nargs, **kwargs)
  File "/home/feedc0de/.local/share/virtualenvs/esphome-1-WzRomffq/lib/python3.12/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
    return self.method(*nargs, **kwargs)
  File "/home/feedc0de/.local/share/virtualenvs/esphome-1-WzRomffq/lib/python3.12/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Util/envs.py", line 242:
    return self.method(*nargs, **kwargs)
  File "/home/feedc0de/.local/share/virtualenvs/esphome-1-WzRomffq/lib/python3.12/site-packages/platformio/builder/tools/piobuild.py", line 342:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 662:
    return method(*args, **kw)
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/feedc0de/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/feedc0de/.platformio/platforms/espressif32@5.4.0/builder/frameworks/espidf.py", line 29:
    import pkg_resources
========================================================================== [FAILED] Took 19.57 seconds ==========================================================================

As my python is to new, I did this.

python -m venv .
bin/pip install pipenv
bin/pipenv --python 3.10
bin/pipenv shell
pip install .

copyrights commented 4 weeks ago

the esphome fork has successfully compiled :)

now upload the firmware through tasmota webinterface or through esphome directly?

tasmota webinterface

0xFEEDC0DE64 commented 4 weeks ago

is dlt.yaml in your example the same yaml file again or is it another already containing the switch configuration?

0xFEEDC0DE64 commented 4 weeks ago

after uploading via tasmota I was greeted with a esphome webinterface, then I tried to flash a yaml file including all the sensor and switch configuration but I get this error during flashing:

$ python -m esphome upload klo-deckenlicht.yaml 
INFO ESPHome 2024.1.0-dev
INFO Reading configuration klo-deckenlicht.yaml...
WARNING The selected ESP-IDF framework version is not the recommended one. If there are connectivity or build issues please remove the manual version.
WARNING The selected ESP-IDF framework version is not the recommended one. If there are connectivity or build issues please remove the manual version.
INFO Detected timezone 'Europe/Vienna'
Found multiple options for uploading, please choose one:
  [1] /dev/ttyACM1 (USB Single Serial)
  [2] Over The Air (klo-deckenlicht.local)
(number): 2
INFO Resolving IP address of klo-deckenlicht.local
INFO  -> 192.168.0.138
INFO Uploading .esphome/build/klo-deckenlicht/.pioenvs/klo-deckenlicht/firmware.bin (1550432 bytes)
ERROR Error binary size: Error: device aborted flash that would have overriden running partition. Check your device log for more information.

0xFEEDC0DE64 commented 4 weeks ago

When doing the upload-factory-ota I get a similar error:

$ python -m esphome -v upload-factory-ota klo-deckenlicht.yaml 
INFO ESPHome 2024.1.0-dev
INFO Reading configuration klo-deckenlicht.yaml...
WARNING The selected ESP-IDF framework version is not the recommended one. If there are connectivity or build issues please remove the manual version.
WARNING The selected ESP-IDF framework version is not the recommended one. If there are connectivity or build issues please remove the manual version.
DEBUG /etc/localtime found
DEBUG 1 found:
 {'/etc/localtime is a symlink to': 'Europe/Vienna'}
INFO Detected timezone 'Europe/Vienna'
DEBUG  -> TZ string CET-1CEST,M3.5.0,M10.5.0/3
INFO Creating a partition table backup.
INFO Resolving IP address of klo-deckenlicht.local
DEBUG Using selector: EpollSelector
INFO  -> 192.168.0.138
DEBUG Features: [2, 3, 4, 5]
DEBUG Partition info: [1, 3, 0, 0, 0, 0, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO Downloading /tmp/tmpmk45zv41 (4096 bytes)
Downloading: [============================================================] 100% Done...

INFO Upload took 1.39 seconds, waiting for result...
DEBUG MD5 of download is 58e615bf83731e87e12725f306d5253d
INFO OTA successful
INFO Resolving IP address of klo-deckenlicht.local
DEBUG Using selector: EpollSelector
INFO  -> 192.168.0.138
DEBUG Features: [2, 3, 4, 5]
INFO Uploading .esphome/build/klo-deckenlicht/.pioenvs/klo-deckenlicht/partitions.bin (3072 bytes)
DEBUG Partition info: [1, 3, 0, 0, 12, 0, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
DEBUG MD5 of upload is e5b569933677bc0b3debb254e74eb9bf
Uploading: [============================================================] 100% Done...

INFO Upload took 0.00 seconds, waiting for result...
INFO OTA successful
INFO Resolving IP address of klo-deckenlicht.local
DEBUG Using selector: EpollSelector
INFO  -> 192.168.0.138
DEBUG Features: [2, 3, 4, 5]
INFO Uploading .esphome/build/klo-deckenlicht/.pioenvs/klo-deckenlicht/firmware.bin (1550432 bytes)
DEBUG Partition info: [1, 4, 0, 23, 168, 96, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 97, 112, 112, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
ERROR Error binary size: Error: device aborted flash that would have overriden running partition. Check your device log for more information.
INFO Trying to restore the partition table backup ...
INFO Resolving IP address of klo-deckenlicht.local
DEBUG Using selector: EpollSelector
INFO  -> 192.168.0.138
DEBUG Features: [2, 3, 4, 5]
INFO Uploading /tmp/tmpmk45zv41 (4096 bytes)
DEBUG Partition info: [1, 3, 0, 0, 16, 0, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
DEBUG MD5 of upload is 58e615bf83731e87e12725f306d5253d
Uploading: [============================================================] 100% Done...

INFO Upload took 0.00 seconds, waiting for result...
INFO OTA successful
WARNING Restore of the backup partition table worked.

copyrights commented 4 weeks ago

what is your device? I only tested this with S1 mini Gen3.

I upload-factory-ota with the simple example. than copy the api key to my productive esphome instance and do the fancy stuff there.

0xFEEDC0DE64 commented 4 weeks ago

Shelly Mini1PMG3

0xFEEDC0DE64 commented 4 weeks ago

So I only have 1 flash after tasmota free, I expanded your yaml file to also include switches and sensors and blueooth_proxy and with this I was able to flash another shelly successful! Just that I cant update remotely is the only problem right now, I just have to come up with a perfect yaml now before I flash all others now :)

copyrights commented 4 weeks ago

Actually I had one S1 with a similar behavior, but I was interrupted during the process and thought the I had mixed something up. So for me the ratio is 1 out of 8 :man_shrugging:.

BuergerJulian commented 3 weeks ago

I tried the same with my Shelly Plus 2PM, the command "esphome upload-factory-ota" is not working.

INFO ESPHome 2024.1.0-dev
INFO Reading configuration wohnzimmer.yaml...
INFO Creating a partition table backup.
INFO Connecting to 192.168.178.149
DEBUG Features: [2, 3, 4, 5]
DEBUG Auth: Nonce is c95adb3f97a00e9789fb1c759b940308
DEBUG Auth: CNonce is 8957b2afd9954bf5fa32156098a92761
DEBUG Auth: Result is 260f8f0321a9984b34f151e80318a5f2
DEBUG Partition info: [1, 3, 0, 0, 0, 0, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
INFO Downloading /tmp/tmpd5i8mlpg (4096 bytes)
Downloading: [============================================================] 100% Done...

INFO Upload took 3.56 seconds, waiting for result...
DEBUG MD5 of download is f912f704f0d827af8a9d88dee9a468df
INFO OTA successful
INFO Connecting to 192.168.178.149
DEBUG Features: [2, 3, 4, 5]
DEBUG Auth: Nonce is 7d1be13f44b52e74e776b4ea71f6fd3b
DEBUG Auth: CNonce is 4bf8c187b2ebe8b4a130dc18de87a86e
DEBUG Auth: Result is 15a6e4b19df7535df1bf2bd62db52ecc
INFO Uploading .esphome/build/shelly-wohnzimmer/.pioenvs/shelly-wohnzimmer/partitions.bin (3072 bytes)
DEBUG Partition info: [1, 3, 0, 0, 12, 0, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
ERROR Error receiving acknowledge binary size: timed out
INFO Trying to restore the partition table backup ...
INFO Connecting to 192.168.178.149
DEBUG Features: [2, 3, 4, 5]
DEBUG Auth: Nonce is 0fcd71c2d0c1b4195bed181547c78940
DEBUG Auth: CNonce is 5157ecfcb677c209951ed602d68a1ee9
DEBUG Auth: Result is 43b9d10529f437d55ed193a28bd1a831
INFO Uploading /tmp/tmpd5i8mlpg (4096 bytes)
DEBUG Partition info: [1, 3, 0, 0, 16, 0, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
ERROR Error receiving acknowledge binary size: timed out
ERROR Restore of the backup failed.

0xFEEDC0DE64 commented 1 week ago

any idea how to remotely update those shellies or do I have to disassemble them now and solder?

esphome / feature-requests

Shelly initial OTA flash #2756