Closed kquinsland closed 2 years ago
probot-esphome[bot]
commented
2 years ago Hey there @jhansche, mind taking a look at this issue as it has been labeled with an integration (bedjet) you are listed as a code owner for? Thanks!
(message by CodeOwnersMention)
jesserockz
commented
2 years ago I have seen this come up before with other components. It seems to happen when a some code (not specifically bedjet) is trying to allocate memory at the same time a BLE event comes in and that memory is also trying to be allocated at the same time, and they clash. The ble stuff is on its own task so is actually happening in parallel.
kquinsland
commented
2 years ago . It seems to happen when a some code (not specifically bedjet) is trying to allocate memory at the same time a BLE event comes in and that memory is also trying to be allocated at the same time, and they clash. The ble stuff is on its own task so is actually happening in parallel.
That would explain some of the crash traces, but the "smaller" traces explicitly mention esp32_ble_tracker::BLEEvent. Unless you mean that "which trace" is just down to which of the two allocations looses the race?
jesserockz
commented
2 years ago . It seems to happen when a some code (not specifically bedjet) is trying to allocate memory at the same time a BLE event comes in and that memory is also trying to be allocated at the same time, and they clash. The ble stuff is on its own task so is actually happening in parallel.
That would explain some of the crash traces, but the "smaller" traces explicitly mention
esp32_ble_tracker::BLEEvent. Unless you mean that "which trace" is just down to which of the two allocations looses the race?
Yeah one randomly loses the race each time.
jhansche
commented
2 years ago I have not seen this myself. Are there any other components for this device, or is it dedicated to the bedjet?
Do you have any additional esphome/board/esp32/framework entries in your config? For example, esp-idf?
Do you know the specs of your board: RAM, flash?
kquinsland
commented
2 years ago Are there any other components for this device, or is it dedicated to the bedjet?
Yes. I don't have easy access to my desktop right now but I can provide a list of the other components that are in use in the next few hours/days. It's the "standard suite" of packages that I include in (almost) every ESPHome config that I do. Off of the top of my head: ntp, webserver, prometheus, OTA FW, status LED, MQTT and the reboot into safe mode switch components.
FWIW, I am using the same module and (almost) the same configuration for some of the xaomi btle sensors: https://esphome.io/components/sensor/xiaomi_ble.html#lywsd03mmc
Do you know the specs of your board: RAM, flash?
It's the "wemos d1 style" module but with the ESP32 module. THe specs @ this link should be accurate but I can confirm once I get back to my computer.
https://doc.riot-os.org/group__boards__esp32__mh-et-live-minikit.html
Is there a way to get ESPHome to dump the full config yaml after all the package/includes are considered?
jhansche
commented
2 years ago So far I've felt that adding the Bluetooth stack alone has been pretty taxing by itself, and the BLEClient contributor (@buxtronix) even warns that BLEClient is resource intensive and you might see instability enabling too many components. I'm actually shocked that this config with both BLE and MQTT will even flash successfully.
Since you said a similar config is used for the Xaomi sensors presumably without this issue, one possible difference with the BedJet component could be the way BLE notifications are triggered. The Xaomi probably sends status updates infrequently (i.e., only when the sensor value changes). While the BedJet firmware sends status updates constantly (every 100-300ms) with no way to back off that frequency. That isn't a problem of the bedjet component itself, but the way the BedJet device works. The component tries to throttle or reduce the load on the board by only processing those incoming notifications once every 5 seconds, rather than processing every incoming notification. However, the notification is still coming in over the Bluetooth stack, so the BLEClient still has to process the incoming packet - we just don't do any additional processing of that buffer outside of that 5 sec interval. This could be contributing to the problem by increasing the likelihood of hitting that race condition that @jesserockz mentioned.
I don't know how we would be able to avoid that, other than profiling and reducing the memory usage of the bedjet component, BLEClient, and/or MQTT client. The BedJet component does try to reduce its memory footprint by for example allocating a single buffer for receiving and processing the incoming status packet and reusing that every time, rather than allocating a new buffer for each status frame. Rapid callbacks like this one (or a frame rendering loop as another example) is really critical to be mindful of memory allocations.
kquinsland
commented
2 years ago Schedule changed / had a spare min to grab the config. Other than the MAC and some comments/notes, this is the config.
I don't have the RAM/ROM free percentages, though. It'll be some time before I can get back to a keyboard / get that.
##
# A simple MQTT <-> BTLE bridget to control the bedjet
# See: https://esphome.io/components/climate/bedjet.html
##
external_components:
- source: github://pr#3490
components: [bedjet]
esphome:
# Note: this will be the hostname that device request during the DHCP dance... but appears to be broken on ESP32 devices
# See: https://github.com/esphome/issues/issues/125
name: ${hostname}
platform: ESP32
board: mhetesp32minikit
# See: https://esphome.io/guides/configuration-types.html#packages
packages:
time: !include ../packages/ntp.yaml
mqtt: !include ../packages/mqtt.yaml
web_server: !include ../packages/web_server.yaml
prometheus: !include ../packages/prometheus.yaml
wifi: !include ../packages/wifi.yaml
status_led: !include ../packages/status_led/mhetesp32minikit.yaml
# We can't use standard OTA package as we have some special automation hooks.
# When an OTA begins, we need to disconnect and disable the bedjet as the BTLE
# connection subsystem is decently resource intensive!
#
# See: https://esphome.io/components/ota.html
##
ota:
password: !secret ota_pass
on_begin:
then:
- logger.log: "Disconnecting clients for OTA update..."
- switch.turn_off: bedjet_78FF_conn
# Enable logging
logger:
level: DEBUG
binary_sensor:
- <<: !include ../packages/binary_sensor/connection_status.config.yaml
button:
- !include ../packages/button/restart.config.yaml
- !include ../packages/button/safemode.config.yaml
sensor:
- <<: !include ../packages/sensor/uptime.config.yaml
- <<: !include ../packages/sensor/wifi_signal_strength.config.yaml
# We need the BTLE subsystem to connect to the bet jet
# See: https://esphome.io/components/ble_client.html
esp32_ble_tracker:
ble_client:
- mac_address: 00:12:34:56:78:ff
id: ble_bedjet_78FF
# We want to expose a switch which will enable/disable the BTLE connection.
# Connection needs to be disabled for OTA and so other $things can control the bed jet (like the OEM apps)
##
# See; https://esphome.io/components/switch/ble_client.html
switch:
- platform: ble_client
id: bedjet_78FF_conn
icon: "mdi:bluetooth-connected"
entity_category: "config"
device_class: "switch"
ble_client_id: ble_bedjet_78FF
name: "Enable BedJet connection"
# See: https://esphome.io/components/climate/index.html
climate:
- platform: bedjet
id: bedjet_78FF
name: "BedJet"
ble_client_id: ble_bedjet_78FF
icon: "mdi:turbine"
# Sets the bedjet clock
# Getting compile time errors: src/main.cpp:507:16: error: 'class esphome::bedjet::Bedjet' has no member named 'set_time_id'
time_id: sntp_time
# See: https://github.com/esphome/issues/issues/3291
visual:
# Not clear to me what unit; i am assuming celsius
##
# Cooling does not let you go below 19
min_temperature: 19
# On the "extra turbo heat" setting, you can go as high as 43c
max_temperature: 43
# Bedjet app does 1c increments
temperature_step: 1I'm actually shocked that this config with both BLE and MQTT will even flash successfully.
I didn't know that BTLE was so taxing. I might be able to do a build tomorrow which will give me the ram/rom used percentages. I don't remember either of them being particularly "full". I'll remove everything but MQTT and the bedjet/btle and add things back until I get an unstable build. That's not going to be a quick test though so results might be a ways out!
Since you said a similar config is used for the Xaomi sensors presumably without this issue, one possible difference with the BedJet component could be the way BLE notifications are triggered.
Yeah, this is sorta what I was thinking. I did some basic RE work on the android app because I was thinking about trying to implement a similar (but way more basic; fan only) ESPhome integration but on the ESP32 inside the bed jet itsself. I've made some (minimal) progress on reverse engineering the ESP32 <-> MCU protocol.
~150ms is ... a lot faster than the xaomi btle things so your explanation seems pretty plausible. I didn't realize that the BTLE stack was "always scheduled"... but it totally makes sense that it would be.
I don't know how we would be able to avoid that, other than profiling and reducing the memory usage of the bedjet component, BLEClient, and/or MQTT client.
Ultimately, that probably would be the solution. But I can turn off most of the "standard config" that I ship in attempt to avoid the problem altogether.
jhansche
commented
2 years ago The latest update from BedJet mentioned that they're planning a new hardware dongle connected to the bedjet's USB port (actually powered serial interface), so the latest firmware should have an active serial interface there. That kind of integration wouldn't need any of the Bluetooth stack which could be a huge win especially for your case where you need a bunch of other stuff.
Another option you could consider could be a second ESP device that just acts as a dedicated bridge between this BedJet component and MQTT 🤔 or possibly another HA-only custom integration that bridges the HA entity <-> MQTT.
But do let us know what you find by limiting the components on the device. I'm still worried about just Bluetooth plus mqtt together, so if it's possible to test without mqtt, that would be helpful as well.
kquinsland
commented
2 years ago The latest update from BedJet mentioned that they're planning a new hardware dongle connected to the bedjet's USB port (actually powered serial interface), so the latest firmware should have an active serial interface there.
Got a link? I'd love to know more about this. What purpose does this dongle solve that the built in ESP can't be used for? Getting a serial port isn't difficult, knowing what to send over it... is. Other than some static analysis I did on the dumped bedjet firmware, I have no documentation about how to tell the MCU to set the fan to 20% and set the heat to 40c.
The only reason I pivoted to RE on the apk was that I had no way to know if i was safely/properly talking to the MCU. BlackBox RE has a place, but not when we're talking about something that could catch fire if done wrong.
If they're going to ship some add on, save us all the trouble and just release some details about how to properly/safely talk to the MCU that runs the fan/heat :). Before too long, we'd have ESPHome/Tasmota working w/ it over Uart (very much like how we do with TuYa today).
where you need a bunch of other stuff. Another option you could consider could beat second ESP device that just acts as a dedicated bridge between the BedJet and mqtt
That's what I thought this component was for and really all I want. I have a "non-trivial" config just because I've slowly built one up over the years and have an appreciation for the standard web UI with a "reboot to safe mode" button and logs and the like. I don't need any of that; it's just in the standard "new project template".
I'm still worried about just Bluetooth plus mqtt together, so if it's possible to test without mqtt, that would be helpful as well.
How are you using this component without MQTT? Unless you're just building your own remote where some GPIO triggers sending a BTLE message to the bedjet or similar duplicationg functionality from the "stock" remote?
jhansche
commented
2 years ago How are you using this component without MQTT?
It becomes a climate entity in Home Assistant. There's no need for MQTT because I use the HA entity to control it.
Got a link? I'd love to know more about this. What purpose does this dongle solve that the built in ESP can't be used for?
Not yet. A month or so back, the CEO of bedjet sent out a Kickstarter update calling for developers to offer their help with the smart-home aspect of the bedjet, because they've tried for about 3+ years to figure it out, and they just can't get it working with the current embedded hardware inside the bedjet. So right now their current direction is a new dongle that will plug into the USB port and acts as a bridge between some kind of smart home integration (they're aiming for Alexa+Google+Homekit) and the device.
I contacted him about that and asked for the serial API details, and offered some ideas, as well as explaining that we have this component available today, over Bluetooth. If he responds something we can use, I'll see if I can work up a serial interface to it. That would actually work with ESP8266, so it would be even cheaper to integrate.
Other than some static analysis I did on the dumped bedjet firmware
Interesting. Would you mind sharing that firmware and what you've found so far? Were you trying to communicate over the USB interface, or did you open it up and try to connect to the components internally?
That's what I thought this component was for and really all I want
That's a fair assumption and I guess a reasonable way to describe it 😄 but what I meant was some other integration like a Node-Red pipeline that just takes the entity from Home Assistant, and pipes that over to MQTT. This component is an integration of bedjet-into-esphome, which is by extension therefore -into-home-assistant. The esphome-to-HA is built-in/automatic. Your goal of getting it into MQTT is on top of that, and it's that extra step that I'm talking about splitting out into another piece of hardware or into HA itself.
kquinsland
commented
2 years ago There's no need for MQTT because I use the HA entity to control it.
Ah. you use the protobuf API not MQTT. Makes sense. I haven't done the leg work to profile/instrument the code (i'm not a C guy) so I can't say for sure... but i'd imagine that there's similar "load" on compute/memory resources with both; we're really only talking about different bytes into a socket. I'd bet that the amount of bytes are roughly similar.
but what I meant was some other integration like a Node-Red
Yep. That's why I don't use the ESPHome protobuf integration with HA. I have some devices that need to play nice with one another even when HA is down; MQTT is supported everywhere!
Your goal of getting it into MQTT is on top of that, and it's that extra step that I'm talking about splitting out into another piece of hardware or into HA itself.
Eh. The HA custom component for ESPHome was built because the mqtt based auto-configuration in HA was neglected for a little while and there was some stuff that you could only do with API. HA has - mostly - fixed this and MQTT for auto discovery is much closer to what you can do with the "native" API now. There's not a ton of difference between the ESPHome native and MQTT above the transport layer.. the ESPHome integration just listens for protobuff and then uses that to manipulate the HA API directly instead of sending a json document to mqtt and letting HA do the API manipulation. /shrug
So right now their current direction is a new dongle that will plug into the USB port and acts as a bridge between some kind of smart home integration (they're aiming for Alexa+Google+Homekit) and the device.
Huh. That... strikes me as a bit odd. I wonder what architectures they have tried/failed with?! The ESP32 has way more than enough horse power to maintain a small connection back to the bedjet servers for remote control and chat with the ANT radio and MCU and - i'd assume - do some BTLE stuff, too. They totally could have the bed just just talking to their servers over MQTT. This is what shelly does with most of their stuff; when you turn off the shelly cloud connection you're given the option to specify your own MQTT server and credentials. Once the device reboots, it really doesn't care / know that it's not talking to the 'official' shelly server anymore. This approach would have been cheap (not super expensive to run a MQTT broker... even @ scale) and would have solved a few birds with a single stone.
Based on (limited, 3rd hand) info, it sounds like they're not willing/able to run their own servers so they're going with the "we'll ship you a basic 802.15 dongle" route and then use the customers existing $hub to do all the control locally. Google and Amazon have both been shipping radios in their hubs for the past several generations for exactly this case... I am uncertain how Bedjet would implement this though as I don't think there's a zwave standard or even a zigbee convention. No idea about Matter, either.
I contacted him about that and asked for the serial API details, and offered some ideas,
I did this about a year ago when I was hoping to save myself some RE time! He politely avoided that part of my email but did ask me if I was interesting in contracting on development around the MQTT stuff... I don't like a lot of things about C/++ so it wouldn't be a good fit.
Would you mind sharing that firmware and what you've found so far? Were you trying to communicate over the USB interface, or did you open it up and try to connect to the components internally?
I don't have a license to distribute the bedjet firmware as it is proprietary/copyrighted. I can say that there is zero readout protection in place so anybody that has a screwdriver and knows how to use esptool.py with a basic USB <-> UART/TTL dongle can get their own copy of the firmware in just a few min. There was a pin header on the ESP daughter-board that was populated with standard dupont 2.54mm pins... so you don't even need a soldering iron for this :). Everything is neatly labeled so there's zero guess work about which pin is GPIO0 ... etc.
The flash is not encrypted and - actually - has a few different firmware images in it! (this took me a few seconds to figure out the first time I looked @ the dump!) The ESP32 can update the firmware on the remote and the MCU as well... images for both are also on the esp32 flash dump :D. strings had some fun stuff in there (pro tip, search for test... always some fun stuff!). I don't have ready access to my notes, but if memory serves:
submcu_server to talk to this IC.Once I dumped the firmware I put the thing back together. Perhaps I should have installed some basic wires on the ESP32 <-> MCU bus (uart0) so I could use sigrock rather than doing everything statically with ghidra. Other than doing a basic "well, I don't see a dedicated USB PHY IC here, so it's probably a uart or i2c port" when I was looking @ the PCB, I didn't probe the USB port. It may be that it is also on uart0. If it is, it might be possible to "inject" commands directly as if the "internal" ESP32 was commanding the 8051 MCU to set the fan to 20%...
I can think of both good and bad reasons to break out the "primary" bus to the front for factory testing so i'd have to go back in with some more time and try to buzz them out to see if the USB port is on uart0 or not.
jhansche
commented
2 years ago I'm only relaying the actual Kickstarter update, which you can find here, with more details about what they've tried and what problems they've encountered: https://www.kickstarter.com/projects/bedjet/bedjet-3-sleep-inducing-climate-control-just-for-y/posts/3063134
It sounds like you know quite a bit about esphome, so your confusion about how I could use a component without MQTT was confusing to me. But equating the MQTT transport to the protobuf API is overly simplistic. The MQTT transport itself is more involved than the protobuf API. There's more state negotiations involved, and the json message format is more verbose than the protobuf messages, which necessarily means more bytes to allocate and transmit. This is why I'm suggesting a test with less stuff involved, due to the BLEClient documentation linked above specifically warning against pairing it with many other components.
Given that we've already established that there's a race condition allocating memory, and that the bedjet's ble status notifications are rapid-fire, I expect that these additional components are likely contributing to the issue. Let's try your test first with MQTT still enabled and see how that fares. If it's still happening, I'd still suggest testing again without MQTT.
I did this about a year ago when I was hoping to save myself some RE time! He politely avoided that part of my email
Sounds very similar to my earlier interactions with him and the beta firmware team 😂 I pressed for a local API that could be integrated into a local solution (at the time I was using SmartThings), and he avoided that part as well.
kquinsland
commented
2 years ago Thanks for the link!
This is the "usage" information for the "full" build (the config posted earlier)
RAM: [== ] 17.0% (used 55732 bytes from 327680 bytes)
Flash: [========= ] 88.3% (used 1620538 bytes from 1835008 bytes)Which almost immediately crashed after booting.
I disabled quite a few things:
And am getting a smaller build:
RAM: [== ] 16.8% (used 55100 bytes from 327680 bytes)
Flash: [========= ] 86.7% (used 1591738 bytes from 1835008 bytes)Which has been a bit more stable but still reliably crashes when I try to interact with the web server.
removing that gives a slightly smaller build:
RAM: [== ] 16.8% (used 54936 bytes from 327680 bytes)
Flash: [======== ] 83.2% (used 1526266 bytes from 1835008 bytes)This did crash immediately after flashing but hasn't since...
I have been able to issue commands over MQTT (via HA) and via the remote. I can see the state being updated not too long after the remote turns the device on.
I re-enabled the connection status and restart/safemode buttons:
RAM: [== ] 16.8% (used 55008 bytes from 327680 bytes)
Flash: [======== ] 83.7% (used 1535538 bytes from 1835008 bytes)That seems to be stable, too. (same "quick" test criteria as above, set state via HA web UI, see if remote shows updated state, change state with remote, see how long it takes HA web UI to update while watching the debug logs to see if there's a crash.)
I re-enabled the status led component:
RAM: [== ] 16.9% (used 55504 bytes from 327680 bytes)
Flash: [======== ] 83.8% (used 1537334 bytes from 1835008 bytes)Still passes the same test so far.
Feeling brave, I enabled the prometheus component:
RAM: [== ] 17.0% (used 55660 bytes from 327680 bytes)
Flash: [========= ] 86.3% (used 1583842 bytes from 1835008 bytes)And I was able to get metrics!
#TYPE esphome_binary_sensor_value GAUGE
#TYPE esphome_binary_sensor_failed GAUGE
esphome_binary_sensor_failed{id="bedjet_controller_status",name="BedJet Controller Status"} 0
esphome_binary_sensor_value{id="bedjet_controller_status",name="BedJet Controller Status"} 1
#TYPE esphome_switch_value GAUGE
#TYPE esphome_switch_failed GAUGE
esphome_switch_failed{id="enable_bedjet_connection",name="Enable BedJet connection"} 0
esphome_switch_value{id="enable_bedjet_connection",name="Enable BedJet connection"} 1Looking at the metrics, there's nothing there that's super useful; I was hoping memory/heap-fragmentation data... I'll go ahead and turn that off.
The uptime and wifi signal sensor are still disabled, but with the web server enabled and the prometheus disabled:
RAM: [== ] 17.0% (used 55676 bytes from 327680 bytes)
Flash: [========= ] 87.5% (used 1605774 bytes from 1835008 bytes)And boom. That's when the instability came back.
And one last "before I have to get to other things" test. This build has INFO level logging with the "full" config minus web/prometheus and the wifi signal strength:
RAM: [== ] 16.9% (used 55536 bytes from 327680 bytes)
Flash: [======== ] 83.5% (used 1532202 bytes from 1835008 bytes)Did get a crash on flash but was able to recover and the usual "quick test" that i've been doing seems to be working.
TL;DR: It looks like the problem was the web server component. I was able to remove that but keep the "rest" of my default config and the firmware has been stable-ish. As soon as the web server comes back into the picture, crashes are probable.
Note: This was all done with the module plugged in via USB. It may be that logging (when done over MQTT) changes the results. Likewise, changing the log level could also change things up.
jhansche
commented
2 years ago Thanks for doing the troubleshooting! Sounds like you've found the two components that appear to be at odds with each other.
I don't know much about the webserver component unfortunately.
kquinsland
commented
2 years ago Sounds like you've found the two components that appear to be at odds with each other.
Yep. I'll close this out. I did a soak test and over the last ~12h the ESP has not crashed once while maintaining a consistent connection to the bedjet. I can live without the web server for now :).
kquinsland
commented
2 years ago I got curious and tried for "one more" component.
I like how the fan speed was added to this dashboard: https://github.com/esphome/esphome/pull/3476#issuecomment-1127012189
But I generally do not like manually creating things in HA; the helpers are a bit of a hack... especially since ESPHome can tell HA to create them and HA will know what to do when the user changes the input automatically.
The docs are not super clear, but with some help from @ssieb on discord, I was able to put this together
# We already have fan speed control via the climate entity but that's buried behind service calls.
# Could manually create an input slider but then it's not accessible on the device page and I have to make more
# automation to link the fan speed to the slider. That's not the automagic way!
# Create a slider so we can directly set the fan speed. This will be part of the device page AND won't require
# manual automation creation.
##
number:
- name: "${friendly_name_short} Fan Speed"
platform: template
id: inp_fan_speed
icon: "mdi:fan"
entity_category: config
unit_of_measurement: "%"
mode: "slider"
min_value: 0
max_value: 100
step: 5
# We'll get a string like `85%` back. Delete the last char, turn into a float and we're golden :)
lambda: |-
std::string speed = id(bedjet_f00d).custom_fan_mode.value();
speed.pop_back();
return atoi(speed.c_str());
# Called when HA sets the number
set_action:
then:
- lambda: |-
auto call = id(bedjet_f00d).make_call();
// We will get a float value, we need to add a % and turn the whole thing into a string
char buffer [4];
sprintf (buffer, "%i%%", (int)x);
//ESP_LOGD("TEST", "WillCall with %s", buffer);
call.set_fan_mode(buffer);
call.perform();
And that's been pretty damn stable :)
jhansche
commented
2 years ago That's great, that's what I've been trying to say: that if we make the data accessible from the component, then it will be much more flexible, allowing custom interactions like what you've done here.
I'm in the middle of refactoring the component into a new "hub" component, so that the climate component is no longer the central communication object. That will allow us to add additional domain implementations, including a fan entity that would do the same thing as what you've done here, and a number entity for things like auto-off. By splitting functionality into separate domains, it also means if you don't ask for that particular component, it won't have to compile that in, which would in theory reduce the size of the image to be flashed. Probably not by much, because all the heavy stuff we still need (BLEClient, ble tracker), but sometimes every little bit helps.
And with that refactor you could add the hub component without adding the climate component, and instead interact entirely with lambdas (what you've done here), through the hub.
kquinsland
commented
2 years ago and a number entity for things like auto-off
hah! How did you guess what my next update to the config was going to be :). It makes more sense for me to drive the timer from HA but
I'll keep my eyes on the esphome release notes to see how the refactor goes.
jhansche
commented
2 years ago Well auto-off is not going to be doable like you did here for fan speed. That's because the code to alter the runtime duration doesn't exist yet. I still need to reverse engineer that command packet format. However, we do already collect the runtime remaining values as it counts down ... but that's not going to be accessible from the lambda yet either.
After the refactor, there will be simple functions exposed by the hub component, that will make both this fan customization, and others like getting/setting auto-off time, much easier to do. From there it's a matter of adding new domain-specific entities in the component, and configuring those in the yaml config.
kquinsland
commented
2 years ago I still need to reverse engineer that command packet format.
I assumed as much.
I was going to implement it with two globals and an interval which calls a lambda. A number input lets the user set the number of 'ticks' to count down and a button to arm/reset the counter. Once armed, every $interval, subtract from the global. When it hits 0, run the "climate off, fan off" code.
It might be easier for me just to do all the timer stuff in HA though as I have other inputs that might cut the timer short.
jhansche
commented
2 years ago @kquinsland out of curiosity, the logs you showed above: were those pulled over the "Logs > Wirelessly" menu from ESPHome dashboard? Or from a USB/UART connection via "Logs > Plug into ..."?
I usually only watch logs over network, because my workstation is not close to the bedroom, and I don't want to keep lugging my laptop into the bedroom in order to watch logs over UART. But one thing I have noticed is that while watching logs (over wifi), the whole ESP device becomes quite unstable, and will drop (and seems to be crashing and restarting, not just disconnecting/reconnecting to the HA API) relatively often. As soon as I stop watching logs, things stabilize considerably.
By any chance could that be what you're seeing? My issue might be completely different from yours, but mine seems to be related to this FAQ entry: https://esphome.io/guides/faq.html#my-node-keeps-reconnecting-randomly which is quite old, has quite a few random possible explanations, and seems to be something we just have to accept.
kquinsland
commented
2 years ago were those pulled over the "Logs > Wirelessly" menu from ESPHome dashboard? Or from a USB/UART connection via "Logs > Plug into ..."?
No. Almost all of my logs came over USB.
I have log level INFO and logs sent via MQTT and have an uptime of a little more than 89,000 seconds. How are you watching the logs w/o web ui? They stream over MQTT when no uart.... I have log level INFO + mqtt logging for the config i published a few posts ago and it's been up just about a day.
jhansche
commented
2 years ago How are you watching the logs w/o web ui?
When I said "over network", I mean over the web UI using the "wirelessly" option. So the logs stream over the API in the web browser.
I do not use MQTT in my config, but I would be worried about the velocity of logs.
But having stability issues while watching the logs makes it really hard to debug the issue. And I wasn't getting anything useful over USB either.
Anyway in my case it was dropping and possibly restarting maybe a couple times a minute while watching logs, but as soon as I stop the logger, it remains stable.
I have to assume that the problem is likely due to the high throughput of the Bluetooth notifications, as while that is doing work, it will be taking up time that would otherwise be used elsewhere, such as in the logger; and vice versa. Maybe the Bluetooth stack hits an overflow of some kind while waiting to deliver the massive stream of incoming notifications (x2 in my case, with dual zone units).
kquinsland
commented
2 years ago ", I mean over the web UI using the "wirelessly" option. So the logs stream over the API in the web browser.
So this is another +1 for "bedjet + webui == not a good idea". I suspect that the instability comes from the web socket server that pushes logs to the web UI as I sometimes would click to adjust the fan speed on the web UI and would get the crash/reboot + stack traces that i posted in the opening messages for this thread. But here's the kicker: every once in a while, the bedjet would get the updated fan speed. This tells me that the crash happened as/just-after the BTLE packet was sent out. This means that the webserver got the POST from the browser and was able to pass that to the component which sent the packet out and produced logs which would have updated the wss.
Somewhere between the bedjet component producing logs and the wss trying to push it to the browser is when the crash would happen... at least for the times when I could adjust the fan via web UI before the crash.
and possibly restarting maybe a couple times a minute while watching logs,
This is part of why I ship an uptime sensor with my 'stock' config. Helps me spot issues like this when I look @ the uptime graphs for a sensor in HA :).
I don't know for sure, but I suspect that as long as the logger component is enabled, all logs are sent to a queue which is then flushed via UART unless/until a MQTT connection can be established. When a wss connection is opened, it appears that the log queue is flushed to both wss and mqtt.
(unless you don't have a MQTT config).
MQTT resource usage has to be lower than webserver + wss so I would use MQTT and then esphome logs .... to get the logs wirelessly. I don't know if esphome supports shipping logs over protobuf.
You'd want to use UART anyways as that should have *all the logs and whatever the ESP bootloader spits out before jumping into the ESPhome binary. Anything that was logged between the last log queue flush and the crash would be lost... and i suspect that the time between flushes is longer for MQTT than it is for UART.
And I wasn't getting anything useful over USB either.
If in doubt, a printf() every few lines or use a few GPIO and toggle them on/off as you enter specific blocks. I don't know enough about platformio / esphome to know if there's some GDB like functionality you could use for enhanced debugging.
kquinsland
commented
2 years ago class esphome::bedjet::BedJetHub' has no member named 'custom_fan_mode'
Use the stable version of bedjet
jhansche
commented
2 years ago Yeah looks like you fixed it, but the error is in your yaml file, not the component:
config/bedjetv3-fb1.yaml: In lambda function:
/config/bedjetv3-fb1.yaml:133:39: error: 'class esphome::bedjet::BedJetHub' has no member named 'custom_fan_mode'
std::string speed = id(bedjet_fb1).custom_fan_mode.value();
^
/config/bedjetv3-fb1.yaml: In lambda function:
/config/bedjetv3-fb1.yaml:141:31: error: 'class esphome::bedjet::BedJetHub' has no member named 'make_call'
auto call = id(bedjet_fb1).make_call();
^The climate methods are only on the climate component, not the hub component. So you can do this by switching to the climate entity id.
However, you could skip the climate component entirely here, and call the new public method on the hub directly: set_fan_index(x), where x is in the range 0-19. Or set_fan_speed(percent) in the range 5-100. The climate component is just going to redirect to this anyway, so it'll save the need to use ClimateCall
jhansche
commented
2 years ago So how does the lambda look like amended?
The actual public functions available now on the BedJetHub component (using the id set in the bedjet: component of your yaml file) are these:
https://github.com/esphome/esphome/pull/3522/files#diff-a5b4d13b378ba30021c2ddf35493c7602db3aaf0e9184e988367fd0d8fa27384R37
Looks like I'm missing a get_fan_speed() here, so the number.lambda has to remain the same for now until that's added.
But for setting the speed, you can replace number.set_action with:
set_action:
then:
- lambda: |-
// `x` is passed in as a float in the range of 0-100, according to your `number` definition
id(bedjet_fb1).set_fan_speed((int) x);Note that a fan speed of "0" is actually not supported, because that would be off 😆 So the range should actually be 5-100 at increments of 5. Or you can make it in the range of 0-19 at increments of 1. Using the 0-19 range, the lambda would call
id(bedjet_fb1).set_fan_index((int) x); // 0-19The benefit of switching to this, is that you don't have to do the string conversion to turn your input float into a "%i%%" string and pass that to the ClimateCall.custom_fan_mode. Because the BedJetHub functions are more machine-friendly - they take unsigned integers:
BedJetHub.set_fan_index(i): i in the range of 0-19, where 0=5%, 19=100%.BedJetHub.set_fan_speed(i): where i is a percent in the range of 5% - 100% at 5% increments.Internally, BedJetClimate will convert the ClimateCall custom fan mode into the index version and actually calls BedJetHub.set_fan_index(). This just cuts out the BedJetClimate component from the equation.
it doesn't update on ESPHome that quickly
So, there are a number of things going on that could contribute to this:
bedjet: section, using update_interval: .... What this interval does is determine how often we pass the last seen (1) BLE notify packet to child components (right now only BedJetClimate).climate: bedjet platform. This will poll the last seen status packet from (1), even if it hasn't been pushed to the child components yet via (1) or (2).As for the "*" of (1): there is one exception to this polling interval, where we added a new "significant change" comparison. In this comparison, we check mode, fan_step, and target_temp_step. If one of those 3 fields changed from the last seen status, we immediately perform the (2) notification.
So based on this, I believe that the "*" note should be triggering an update of the Climate component as soon as the notification packet is received with a different fan speed (i.e., even if you changed it from the remote).
If you have verbose logging enabled, that could help identify if something is missing. We do also have some debug logs. Some example logs to look for, whenever the Climate component updates:
[D] update()[D] update_status result=<true|false>[V] Handling on_status with data=...[V] After on_status, new mode=...With the "significant change" comparison, you should be seeing those^ log message appear as soon as the status changes from the BedJet's BLE notification. If you don't see that log as soon as the state changes (within 1-2 seconds max), then it indicates something is not working properly.
Another option might be to try dropping one or more of the update_interval settings mentioned above. But my opinion is that it really shouldn't be necessary.
jhansche
commented
2 years ago So are you saying you change the number slider, it sets the speed on the BedJet as expected, but then the slider reverts back to the previous value? And then some unknown time later, the slider updates again back to the correct value that you changed it to?
Or is this delay only when you change the fan speed from the BedJet remote?
jhansche
commented
2 years ago According to Template number, the lambda is only called once per update_interval:
lambda (Optional, lambda): Lambda to be evaluated every update interval to get the current value of the number.
So you're only going to get the current value (via this atoi(speed.c_str()) call in your template number lambda), once per 60 seconds, as per the template number default update_interval.
The problem is that this template doesn't know how to get immediate updates from the BedJet. So your only way of updating the slider, is with that update_interval. If you drop that down to ~5 seconds, then it will poll the BedJetClimate.custom_fan_mode.value() every 5 seconds.
jhansche
commented
2 years ago More importantly, I think, is that eventually we'll be able to add a bedjet platform number component, which will be able to expose the fan speed, in both directions: it'll be able to update as soon as you change it with the Remote, and also will be able to notify BedJet when to change the speed when you change the slider. And in fact, we could use the fan component for this, rather than number (I don't think number makes sense for the Fan speed).
Your current template number is just a workaround for the fact that we haven't added that child component to the hub yet.
jhansche
commented
2 years ago The reason your helpers+automations approach is instant, is because it reacts to the BedJetClimate's state change. The ESPHome template number does not "react" to anything - it can only get an updated number on the update_interval, which defaults to 60sec, and since you don't override that in your config, 60s is what you get. That explains why it takes so long for your template number to update to the current value.
If instant updates in HA when you change the speed from outside of HA is important to you (I'm not sure why this would have to be "realtime", given the whole use case of BedJet itself), then your only options for now are:
update_interval.Cool, is that coming soon?
It has to be built on top of the BedJetHub functionality, which is still incomplete. I don't want to add that into the Hub PR for now, because I think that would expand the scope of the PR, which is already a pretty major rewrite. I could always make a separate experimental branch that you'd be able to use in your custom_components just like you are now, and that would work. But I need to prioritize figuring out the stability issues of the Hub first.
Recently, I switched my controller to use the esp-idf framework, and I think that really improved a lot (seriously), but there are still some things that just aren't working right, and I can't yet rule out if the issues are caused by the rewrite. That's why the PR is in "Draft" state right now.
Lemme just say
Thanks for the sentiment; though I'm not angry with them or anything - I'm just glad that I was able to find a way to make it work! I agree they've dropped the ball here, big time. I'm currently in a pretty deep back and forth with the CEO Mark right now, about suggestions for moving forward with a smart home solution - which it sounds like they are still committed to delivering, even if they can't do it with the onboard controller. I'm slowly trying to chip away, and I've just about convinced him that it's a better idea to abandon the Alexa-only solution they're currently exploring, in favor of a Matter solution which will be compatible with Alexa+Google+Homekit (and any other Matter controller). If they can ditch the entire BLE GATT server as well, it might even be able to fit in the onboard controller, and then they can convert the BedJet apps to act as a Matter controller rather than controlling it over BLE. 😎
Easier for you to watch what happens in real time
Oh, I see... So the slider works, and sets the BedJet fan near-instantly? But the actual % number to the right is delayed. Is that right? I do think the update_interval is the problem here, because the BedJet has no way to "notify" the template number of a state change. That's why your automation+helper solution worked more instantly, because it is reacting to the Climate entity's fan mode changes. You can also make an automation that triggers the update without having to wait for the update_interval. Something like:
homeassistant.update_entity('number.inp_fan_speed')You can test if that would work, by doing the same test you did in the video, and then use the Developer Tools section to call that update_entity service manually. If it works, then your number entity should update as soon as you call the service.
jhansche
commented
2 years ago You still have the update_interval option. If you don't mind that the number component will be asking the BedJetClimate device every N seconds "what's your fan speed now?", then you can absolutely drop that interval to as low as you want to go. Asking for the current custom_fan_speed value should be instant because it's all happening in-memory only on the ESP32 - it doesn't have to make a round-trip to HA or over BLE to the BedJet, it's just going to ask what the current state of BedJetClimate is - and since both the Template Number and the BedJetClimate are components executing on the same ESP32, it's not making any external requests.
So if you want to set that to update_interval: 1s, then it will update every second. All you would do is add it to the number config:
number:
- name: "BedJet FB1 Fan Speed"
platform: template
id: inp_fan_speed
icon: "mdi:fan"
entity_category: config
+ # or something else acceptable for you (2s, 5s, etc)
+ update_interval: 1sThe downside is that it has to do all that string manipulation 😞 So you'll get the custom_fan_mode as a string, strip off the %, and then use atoi to convert that string to an integer.
kquinsland
commented
2 years ago I'm slowly trying to chip away, and I've just about convinced him that it's a better idea to abandon the Alexa-only solution they're currently exploring, in favor of a Matter solution which will be compatible with Alexa+Google+Homekit (and any other Matter controller)
I wish that was a more open conversation. Alexa only means they're going one of two ways:
I presume they're going with 802.15 as running your own servers is a perpetual cost and a privacy/reliability liability. Since it sounds like the built in ESP32 does not have enough resources to run much more additional software, it sounds like they're going to use the UART via the USB port on the front and ship a bolt on.
Matter is still a ways out and that's going to be a tough decision as bedjet 'cloud' connectivity is already delayed... although I can't imagine they're going to have an easy time getting buckets of any 802.15 chips now. I am looking at multi month lead-times on even some mosfets... let alone micro-controllers and other ICs!.
Although, Matter does support WiFi so it might be feasible to just ship Matter compatibility to the ESP32 with a firmware update. This is what the HA people are showing off later this month: https://www.home-assistant.io/blog/2022/05/29/matter-in-home-assistant-workshop-announcement/. Apparently HomeAssistant will be able ot provision an ESP32 over BTLE.
(i presume. I know little about Matter other than it's a collection of technologies... including IPv6. Presumably whatever the "central coordinating entity" in a Matter network is will be able to work with 802.15 and 802.11 PHY)
Regardless of what they go with, it will be easier to sniff / RE the dongle <-> ESP32 comms. Unless they do something complicated to secure the comms between the dongle and the ESP32, it should be reasonably easy to reverse engineer and then build a proper ESPHome dongle.
jhansche
commented
2 years ago TCP. This is how "most" Alexa things work: servers in the nearest AWS POP figure out you're trying to turn on a light that is managed by - for example - the Hue integration. Server reaches over to Hue servers which then send a signal back down to your local bridge through the persistent TCP connection that the bridge keeps with the hue servers.
The plan (as of 6 days ago) was to use this: https://www.espressif.com/en/solutions/device-connectivity/ack-solution by putting the ACK SDK onto a new ESP32 USB dongle that would plug into the USB port on the BedJet, which will (soon) have a USB-Serial API to bridge from the new dongle<->BedJet.
it sounds like they're going to use the UART via the USB port on the front and ship a bolt on.
exactly.
Matter is still a ways out and that's going to be a tough decision as bedjet 'cloud' connectivity is already delayed...
This is actually my argument for Matter.
Regardless of what they go with, it will be easier to sniff / RE the dongle <-> ESP32 comms. Unless they do something complicated to secure the comms between the dongle and the ESP32, it should be reasonably easy to reverse engineer and then build a proper ESPHome dongle.
No it won't be secured, and it won't need to be sniffed or RE'd. I'm awaiting the USB-Serial API specs to test it out. However, I don't want to go in that direction personally, because being a dual zone setup, I can only plug my ESP32 into 1 device at a time :) So I actually prefer the over-the-air approach, because I can control both BedJets with a single ESP right now. Yes it sounds appealing to be able to plug an ESP right into the USB port and have instant control - but that only works with a single unit.
kquinsland
commented
2 years ago I wish I shared your enthusiasm about Matter. I am very curious about it and I really want everybody to be on the same page but this just has XKCD 927 written all over it.
they are already delayed (3 years), so what's another couple months?
I get this argument but from the perspective of a business, Matter is already delayed so it's unwise to assume that it will be ready in a few months / silicon shortages + now you have to figure out how to implement it.
Why spend the time on ACK for Amazon-only, when you can spend that time on Matter instead
Because nobody (that I am aware of) is selling a devkit / SDK that has most of the batteries included for Matter. The ACK route is 75% complete from the start, just plug in your AWS account details and hire somebody to spin you a new PCB / case.
If Espressif had a reference Matter implementation available then I could see that being the obvious choice over ACK modules.
No it won't be secured, and it won't need to be sniffed or RE'd. I'm awaiting the USB-Serial API specs to test it out.
Are those under NDA or can you share if/when you get them?
However, I don't want to go in that direction personally, because being a dual zone setup, I can only plug my ESP32 into 1 device at a time :)
A single ESP32 module has 3 UARTs. 2 of which are usable on most modules (IIRC a few pins used for external flash conflict with one of the uarts but i could be wrong about this). You absolutely could have two bedjets controlled via a single ESP32 module.
jhansche
commented
2 years ago If Espressif had a reference Matter implementation available then I could see that being the obvious choice over ACK modules.
They do, for over a year:
Are those under NDA or can you share if/when you get them?
I have not been asked to sign anything, or been instructed not to share them. Unless I'm instructed otherwise when delivered, I will assume that there will be no restrictions.
A single ESP32 module has 3 UARTs.
Well sure, but I also don't want to plug a single ESP into 2 BedJets 😆
mwolter805
commented
2 years ago Extremely interested in the USB-Serial API! Would love to use the USB port on the BedJet to connect an ESP via ethernet. 😀
kquinsland
commented
2 years ago They do, for over a year:
Huh. Well TIL. Thanks for the links. I guess i wasn't looking that hard / not seeing a ton of technical details about it made me think that it was all still very much "in development".
jhansche
commented
2 years ago The protocol and data models look pretty well fleshed out. There's a ton of info and links at https://github.com/esphome/feature-requests/issues/1430 as well, if you want to follow the ESPHome FR.
kquinsland
commented
2 years ago The protocol and data models look pretty well fleshed out. There's a ton of info and links at esphome/feature-requests#1430 as well, if you want to follow the ESPHome FR.
This also got me caught up quickly. TY for the links!
jhansche
commented
2 years ago my concern was causing processing overhead on the ESP
To be fair, the string manipulation is going to add processing overhead, but I don't think it would be so much that it causes actual problems. Once I add the get_fan_speed function to the hub, you can similarly replace your climate.custom_fan_mode manipulation with a simple getting function.
and flooding the network/HA with requests
Also looking through the code, it appears that publishing state actually does make an api request to HA - at least I can find no code that compares last-known state to new state and decides not to send it if it didn't change. By tracing through the publish_state() functions, it looks like it just sends that state every time it's called. And Template Number also does no comparison of previous state before publishing:
void TemplateNumber::update() {
if (!this->f_.has_value())
return;
auto val = (*this->f_)();
if (!val.has_value())
return;
this->publish_state(*val);
}So while the process of obtaining the value from BedJetClimate is all local, it looks like the process of publishing that value back to HA will make a round trip request, even if it didn't change 😞
The problem
I am experiencing some odd but often crashes when using the BedJet climate component.
Looking at the stack trace there seem to be two types of crashes; at least some stacktraces are much shorter than the others.
I can't figure out a reliable trigger for this crash: I can sometimes trigger it via HA --[mqtt]--> esp and other times it happens all on it's own with no trigger from me: the BTLE device it's controlling is static, no commands from esphome web UI or HA to adjust the state.
When I was able to observe a crash immediately after some sort of control/input, i have included what I did.
Which version of ESPHome has the issue?
Version: 2022.5.0
What type of installation are you using?
pip
Which version of Home Assistant has the issue?
Home Assistant Core 2022.5.3
What platform are you using?
ESP32
Board
mhetesp32minikit
Component causing the issue
bedjet
Example YAML snippet
Anything in the logs that might be useful for us?
(Note: I have replaced the other nearby BTLE mac addresses with placeholders
DEADBEEF000N. I wanted to provide some additional context to the crashes so I kept the surounding log lines but modified them to strip PII)This happened right after I did this in HA;
I did NOTHING other than use the physical remote to set the bed jet from OFF to COOL (10% fan) and waited the <30s for the polling to occur. About 20s later, this is the output:
And another crash with the same trigger as first one (calling the service via HA web UI)
And after using the HA web UI to set fan speed to 20%
Additional information
No response