Closed glbailey closed 9 months ago
A compiler Segfaulting is often caused by memory issues. Have you tried replicating these issues on another (non virtual) system?
This is an "internal compiler error", so not the usual out of memory crash. Given the random nature and weird garbage, I would suspect either RAM or disk data corruption.
In the devcontainer
for ESPHome I regularly get these errors thrown at random where it complains about being unable to allocate memory:
In file included from /esphome/.temp/platformio/packages/toolchain-xtensa-esp32/xtensa-esp32-elf/include/c++/8.4.0/string:40,
from /esphome/.temp/platformio/packages/toolchain-xtensa-esp32/xtensa-esp32-elf/include/c++/8.4.0/stdexcept:39,
from /esphome/.temp/platformio/packages/toolchain-xtensa-esp32/xtensa-esp32-elf/include/c++/8.4.0/array:39,
from /esphome/.temp/platformio/packages/toolchain-xtensa-esp32/xtensa-esp32-elf/include/c++/8.4.0/tuple:39,
from /esphome/.temp/platformio/packages/toolchain-xtensa-esp32/xtensa-esp32-elf/include/c++/8.4.0/functional:54,
from src/esphome/core/helpers.h:5,
from src/esphome/components/esp32_bt_classic/utils.h:4,
from src/esphome/components/esp32_bt_classic/utils.cpp:3:
/esphome/.temp/platformio/packages/toolchain-xtensa-esp32/xtensa-esp32-elf/include/c++/8.4.0/bits/char_traits.h:40:10: fatal error: .piolibdeps/esp32-office-bt-monitor/libsodium/libsodium/src/libsodium/bits/postypes.h: Cannot allocate memory
#include <bits/postypes.h> // For streampos
This occurs even though there is enough system memory available. I reccon this is caused by the docker container being a Linux environment, and thus having to run virtualized on my Windows host machine. This is an issue in the virtualization layer, and not something ESPHome can do anything about.
Though not exactly the same as the issue @glbailey describes, I guess the root cause is the same.
Hi. Just want to check in to say that I am seeing the same behavior (random failures and core dumps while compiling ESPHome firmwares, and pressing retry multiple times to make it work).
I am running HA insid a Virtual Box on Win10. I am yet to try solving this on my own...
something between the Home Assistant Docker and VirtualBox is having trouble managing memory (?)
If that's true, then I wonder why the rest of the Home Assistant is running fine.
Compiling is very stressful on everything. It's a good way to find issues. :grin:
Yes, I wonder the same thing, but I could not find a solution to fix that directly.
Removing ESPHome from HA and instead running it in a separate Docker container has resulted in perfect compiles every time with no other issues.
It’s a little less convenient for getting update notifications but I have yet to find a downside to running it separately as far as devices are concerned.
It is less secure though as ESPHome doesn’t have a login/password mechanism (that I have run across).
On Jun 30, 2023, at 12:45 AM, u17194907425 @.***> wrote:
Hi. Just want to check in to say that I am seeing the same behavior (random failures and core dumps while compiling ESPHome firmwares, and pressing retry multiple times to make it work).
I am running HA insid a Virtual Box on Win10. I am yet to try solving this on my own...
something between the Home Assistant Docker and VirtualBox is having trouble managing memory (?)
If that's true, then I wonder why the rest of the Home Assistant is running fine.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
If that's true, then I wonder why the rest of the Home Assistant is running fine.
When compiling complete Esphome firmware, there are a lot of files read and written and this would cause a lot of memory access rapid succession. Home Assistant's memory access pattern is much more 'relaxed' in that sense.
It is less secure though as ESPHome doesn’t have a login/password mechanism (that I have run across).
Seeing as you run the Esphome container in docker already, you could set it up behind a reverse proxy with an autentication layer. It is definately not as easy as just exposing Esphome to your network, but the options are there if you are willing to tinker with it.
set it up behind a reverse proxy with an autentication layer.
Ah, thank you for that! I shall dive down that rabbit hole next :-)
On Jul 1, 2023, at 2:55 AM, RoboMagus @.***> wrote:
It is less secure though as ESPHome doesn’t have a login/password mechanism (that I have run across).
Seeing as you run the Esphome container in docker already, you could set it up behind a reverse proxy with an autentication layer. It is definately not as easy as just exposing Esphome to your network, but the options are there if you are willing to tinker with it.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
Just wanted to check in and confirm that the described behavior still exists. Yesterday I had to retry compilation 6 or 7 times before it was successful.
ESPHome 2023.10.1 Home Assistant 2023.9.2 Supervisor 2023.10.0 Operating System 11.0 VirtualBox 6
Yes, I gave up on trying to make it work within Home Assistant which is running in virtual box. Instead, I moved ESPHome to a docker container on the main machine (outside of virtual box). Have not had an issue ever since. So I think the issue is memory management related to Virtual Box but I don’t have the tools or knowhow to dig deeper. All the devices work fine in HA with this configuration and whenever I need to work on or update a device I just start up ESPHome in the docker container.
On Oct 23, 2023, at 1:16 AM, u17194907425 @.***> wrote:
Just wanted to check in and confirm that the described behavior still exists. Yesterday I had to retry compilation 6 or 7 times before it was successful.
ESPHome 2023.10.1 Home Assistant 2023.9.2 Supervisor 2023.10.0 Operating System 11.0 VirtualBox 6
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
@glbailey thanks for sharing. BTW since you have mentioned VirtualBox, which is a valid possible root cause of the issue... Are you on version 6 or have you migrated to v7 already? Coz if you are on 6 there is possibility that the issue is gone on VB7
Well now that’s a very good point - I’m on 6.1 and haven’t checked for updates in a while. Guess I’m going to have to do some testing to find out.
On Oct 23, 2023, at 9:45 AM, u17194907425 @.***> wrote:
@.***(https://github.com/glbailey) thanks for sharing. BTW since you have mentioned VirtualBox, which is a valid possible root cause of the issue... Are you on version 6 or have you migrated to v7 already? Coz if you are on 6 there is possibility that the issue is gone on VB7
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
I had similar behavior to ESPhome. In my case, the culprit was my antivirus program. After turning it off, ESPhome operation returned to normal.
Some updade on my observations: The issue with Multiple compile retries to successful compile has completely gone right after I changed RAM on my virtual instanve of VirtualBox from 4 to 8 GB. Now even quite long compilations are stable again. I do not have to press "retry" any more.
I'm using docker and set shm size to 500mb from 64mb and still needing to do multiple retry for install (compile, and connect) as well as view logs (connect)
@mastersf you're mentioning multiple things. Only the compiling is relevant here. And actually, I'm going to close this because it's a VirtualBox issue. If you have other issues, come ask on discord.
The problem
Attempts to compile and upload for each device takes 8 to 10 retry attempts before succeeding, with seemingly random errors at each retry. No YAML or other changes made at each retry, but eventually compile succeeds. Note this example is 1 of 12 different devices being used in this system including multiple Sonoff S31 devices, ESP8266 D1 mini, eSP32Cam and ESP32dev devices using esp-idf and arduino platforms. Compilation failures affect ALL of these devices and platforms. No obvious device dependencies. Restarting of ESPHome add-on sometimes shortens the number of retries before success and a full reboot of Home Assistant often allows first time compile success for 3 to 4 devices before returning to multiple failed compilations. This issue is not new; has been occurring since Home Assistant versions back to at least October 2022. I have increased cores from 2 to 4 and increased memory from 3GB to 6GB with no change in (seemingly) random compile errors. Using "clean build files" before a retry seems to make no difference. Attached are results from a series of 8 retries before we got a successful compile and upload showing the errors at each step.
Which version of ESPHome has the issue?
2023.3.2
What type of installation are you using?
Home Assistant Add-on
Which version of Home Assistant has the issue?
2023.3.5
What platform are you using?
ESP32-IDF
Board
ESP32 DEVKIT V1
Component causing the issue
compiler
Example YAML snippet
Anything in the logs that might be useful for us?
Additional information
8 attempts to compile, 7 failures. NO changes to yaml file between attempts.
System specifics:
Linux Mint 20.3 Cinnamon running VirtualBox 6.1.38_Ubuntu on 8 core i9-9880 16G Ram 512G SSD HA virtual machine assigned 4 processors: core processor usage 2%, supervisor processor usage 0% HA virtual machine assigned 6GB memory - core memory usage 10.5%, supervisor memory usage 11.1%
Home Assistant: Version core-2023.3.5 Installation Type Home Assistant OS Python Version 3.10.10 Operating System Family Linux Operating System Version 5.15.90 CPU Architecture x86_64
ESPHome: version 2023.3.2 Add-on CPU Usage: 0% Add-on RAM Usage: 11.9%
(see hot-tub-monitor.yaml for actual yaml file being compiled for a standard ESP32dev board.)
Note this is 1 of 12 different devices being used in this system including multiple Sonoff S31 devices, ESP8266 D1 mini, eSP32Cam and ESP32dev devices using esp-idf and arduino platforms. Compilation failures affect ALL of these devices. Restarting of ESPHome add-on sometimes shortens the number of retries before success and a full reboot of Home Assistant often allows first time compile success for 3 to 4 devices before returning to multiple failed compilations. This issue is not new; has been occurring since Home Assistant versions back to at least October 2022. I have increased cores from 2 to 4 and increased memory from 3GB to 6GB with no change in (seemingly) random compile errors.