Closed AleXSR700 closed 1 month ago
@btsimonh Can you please have a look?
It is working fine in 14.0.0.1
@Jason2866 - family issues, but I'll try to take a look :(
I built tasmota32s3-bluetooth with use config overrides of:
#define WEBCAM_DEV_DEBUG
#define BLE_ESP32_DEBUG
#define USE_MI_ESP32
#define USE_BLE_ESP32
#define USE_EQ3_ESP32
works on fresh build of 14.2.0.1 on S3. EQ3 polling all good, out of 10 polls of three devices, only one failure (to be expected). works on build of 14.2.0.5..... I note that 1248bd138c7eda34fe91d9f60721a78210a27b56 changes the core version (commit before 14.2.0.6) works with 14.3.0.1 69f7f155dd3750ff44f620556d3d46c3ae25beda But... possibly more failed to connect, a few in a row.
failed to connect to device low level rc 0xd
...
failed to connect to device low level rc 0x216
but probably not more than expected with Eq3.... (it's a crap device ref BLE coms).
Added a battery to a temp/hum sensor (old ATC firmware). The adverts are being received, and the device displayed.
Conclusion: On my own build of latest dev, it seems working on s3. I do NOT have any aliases defined.
sample logging of a transaction with a distant eq3:
21:11:49.358 EQ3: 001A22085B1E: Op queued len now 1
21:11:49.360 EQ3: 001A22085B1E: Op dequeued len now 0
21:12:08.484 WIF: Checking connection...
21:12:19.425 BLE: failed to connect to device low level rc 0xd
21:12:19.428 BLE: failed to connect to device
21:12:19.549 BLE: BLETask: op failed -11
21:12:20.358 EQ3: 001A22085B1E: trv operation failed - retrying -11
21:12:20.362 RSL: BLE = {"BLEOperation":{"opid":"37","stat":"-11","state":"FAILCONNECT","MAC":"001A22085B1E","u":"128","svc":"3e135142-654f-9090-134a-a6ff5bb77046","char":"3fa4585a-ce4a-3bad-db4b-b8df8179ea09","notifychar":"d0e8434d-cd29-0996-af41-6c90f4e0eb2a","write":"03180A0F150B31000000000000000000"}}
21:12:28.483 WIF: Checking connection...
21:12:34.358 M32: Kick off readOneSensor
21:12:34.360 M32: Kick off tele sending
21:12:35.359 RSL: SENSOR = {"Time":"2024-10-15T21:12:35","ATC2682c3":{"mac":"a4c1382682c3","Temperature":23.2,"Humidity":51.2,"DewPoint":12.5,"Btn":0,"Battery":95,"RSSI":-50},"TempUnit":"C"}
21:12:43.356 BRY: GC from 5690 to 3707 bytes, objects freed 7/41 (in 0 ms) - slots from 46/91 to 45/91
21:12:43.362 RSL: STATE = {"Time":"2024-10-15T21:12:43","Uptime":"0T00:19:40","UptimeSec":1180,"Heap":110,"SleepMode":"Dynamic","Sleep":1,"LoadAvg":999,"MqttCount":0,"Berry":{"HeapUsed":3,"Objects":41},"POWER1":"OFF","POWER2":"OFF","Dimmer":19,"Color":"303030","HSBColor":"0,0,19","Channel":[19,19,19],"Scheme":0,"Width":1,"Fade":"OFF","Speed":1,"LedTable":"ON","Wifi":{"AP":1,"SSId":"MYSSID","BSSId":"AC:B6:87:C7:0E:B8","Channel":11,"Mode":"HT40","RSSI":100,"Signal":-33,"LinkCount":1,"Downtime":"0T00:00:04"}}
21:12:43.416 RSL: BLE = {"Time":"2024-10-15T21:12:43","BLEDevices":{"total":4,"001A22092CDB":{"i":0,"r":-72},"001A22092C9A":{"i":1,"r":-71},"001A22085B1E":{"i":2,"r":-88},"A4C1382682C3":{"i":3,"r":-50}}}
21:12:43.431 RSL: BLE = {"Time":"2024-10-15T21:12:43","BLE":{"scans":59,"adverts":4736,"devices":4,"resets":0}}
21:12:45.360 RSL: 001A22085B1E = {"cmd":"poll","result":"ok","MAC":"001A22085B1E","tas":"tasmota-9CE64C-1612","RSSI":-88,"stattime":1729023165,"temp":17.0,"posn":0,"mode":"manual","hassmode":"off","boost":"inactive","dst":"set","window":"closed","state":"unlocked","battery":"GOOD"}
21:12:45.384 RSL: BLE = {"BLEOperation":{"opid":"38","stat":"7","state":"DONENOTIFIED","MAC":"001A22085B1E","u":"128","svc":"3e135142-654f-9090-134a-a6ff5bb77046","char":"3fa4585a-ce4a-3bad-db4b-b8df8179ea09","notifychar":"d0e8434d-cd29-0996-af41-6c90f4e0eb2a","write":"03180A0F150B31000000000000000000","notify":"020109000422"}}
@AleXSR700 could you try again? If with 14.3.0.1 you still have consistent failures, check your wifi traffic (including MQTT, etc.). I had none beyond the GUI. Wifi and BLE do NOT play nice.... If possible, check serial logs with all connections closed to the esp. If necessary, I will build for 'std' esp32 (I have some webcam boards I bought for Eq3 use because of external antennas), and test again; just not sure when.
I will start a new build and start testing tomorrow.
Btw, you are building the base binaries instead of the bluetooth
variant?
#define USE_MI_ESP32
#define USE_BLE_ESP32
#define USE_EQ3_ESP32
These are all part of the bluetooth variant (not sure if there is more to it than this plus the libs).
I will test as before with the bluetooth
variant and if that fails, build and test the base image with those three enabled manually to countercheck.
I built tasmota32s3-bluetooth
with the above. (bluetooth variants did not used to have USE_BLE_ESP32 or USE_EQ3_ESP32. now it seems they all do?)
I double checked configs, and it seems FIRMWARE_BLUETOOTH contains all for EQ3 except in _mi32 variants.
so double check your platformio_tasmota_cenv.ini
is the same as the platformio_tasmota_cenv_sample.ini
(for your c3), and the pure bluetooth variants should be fine. The fact that you have logs from the EQ3 driver indicate your build contained everything required.
I think the Mi32 is not compatible with the new bluetooth anyway. So if you build with #define USE_MI_ESP32, then I think you are using legacy build which is not the same as the bluetooth variant. https://tasmota.github.io/docs/Bluetooth_MI32/
But I am not sure what happens when you build - blueooth
variant but enable mi32-legacy
mode. I thought the latter does not use the latest BLE modules or something like that.
I didn't think EQ3 was even compatible with Mi32-legacy mode.
https://discord.com/channels/479389167382691863/796060106776117288/1216387713574637641 and following.
I built esp32c3bluetooth (without defining any of the above USE*) and am still seeing the same issues in 14.3.0.1 (bluetooth).
I don't have a c3 to test with :(. could you try on plain esp32?
There is a big update ahead for the BLE legacy stuff. Look in open PRs ;-) There is no TRV driver for the MI BLE (and there will be none). The MI BLE is the more actual BLE code.
I don't have a c3 to test with :(. could you try on plain esp32?
Yes, I have a devkit esp32. So if you have a build you would like me to test, happy to.
There is a big update ahead for the BLE legacy stuff. Look in open PRs ;-) There is no TRV driver for the MI BLE (and there will be none). The MI BLE is the more actual BLE code.
Would kind of be nice to have one BLE variant for all BLE devices :)
Only for information: I just tested the latest dev 14.3.0.1 on a "plain" ESP32 and it is working without problems.
Would kind of be nice to have one BLE variant for all BLE devices :)
Will not happen since the drivers have different goals and design concepts.
looking at the berry in the PR, seems an EQ3 driver could be written in berry over the 'new' legacy stuff? (I am not volunteering!).
@btsimonh @SteWers : Did you test with latest TRV firmware 1.46 or above?
I now flashed the tasmota32-bluetooth and tasmota32c3-bluetooth from here: https://github.com/tasmota/install/releases/tag/2024-10-16.d.e39f1cc83f17100adc41fb8b6d18a792ecd50c11
I wanted to rule out any build issues.
Neither the ESP32 nor the ESP32C3 seem to work properly. In the WebUI I see all my ATC information (Xiaomi Mijia devices) correctly but all my EQ3s only show 0 values (except for RSSI, which is not a value published by the EQ3).
So all set point temperatures, valve positions and batteries are 0 all the time.
I flashed the downloaded binaries and then performed a reset 1. Then connected back to network, turned on BLE and power cycled.
Did you test with latest TRV firmware 1.46 or above?
I don't know the firmware of the TRV.
I don't know the firmware of the TRV.
It is displayed when you put the batteries in. In the newer firmware (1.46 and above) the manufacturer changed the pairing mechanism/key. https://github.com/arendst/Tasmota/issues/16775
So if the tested EQ3 firmware is older, the issue might be related to/the same as https://github.com/arendst/Tasmota/issues/16775 and would not show up with older devices.
I upgraded of mine to 1.46, so it is not possible to test if it would work with the older ones.
It is displayed when you put the batteries in.
Mine has version 1.20
mine are all older firmware. it's odd that is works for you with older TAS, but newer EQ3 firmware - the key stuff was some time ago. I'd suggest that the key stuff was not about the key (I don't think it needed it in the end? it just worked again after some tweaks?), but about doing some handshake. Maybe check the commits associated with https://github.com/arendst/Tasmota/issues/16775 and see what was added to make it work then, then check for changes in the idf/nimble?
mine are all older firmware. it's odd that is works for you with older TAS, but newer EQ3 firmware - the key stuff was some time ago.
No, I deliberately stayed with firmware versions newer than that handshake/pairing change. So firmware v12 and v13 versions post https://github.com/arendst/Tasmota/issues/16775
Maybe check the commits associated with https://github.com/arendst/Tasmota/issues/16775 and see what was added to make it work then, then check for changes in the idf/nimble?
Since all of your devices are older firmware but work and the same applies for SteWers, I was thinking the same thing. Seems like a change made back im v12 to fix the compatibility with newer EQ3 firmware is no longer present or working in v14.
Maybe @h2zero has an idea?
looked for my 'spare' eq3 today (the one i dismantled and broke the wheel on...) to see if I could upgrade it and test. But it's not going to happen: Washing has so far not brought it to life....
ok, that one is fubar. (It does now start, but I can't read the display). I found another... (one which had broken off the rad). It's now on 1.46, and I get at a distance of 20cm:
15:44:09.723 EQ3: 001A22092FB7: Op queued len now 1
15:44:09.725 EQ3: 001A22092FB7: Op dequeued len now 0
15:44:12.314 BLE: failed subscribe for notify
15:44:12.438 BLE: BLETask: op failed -12
15:44:13.691 EQ3: 001A22092FB7: trv operation failed - retrying -12
15:44:13.697 RSL: BLE = {"BLEOperation":{"opid":"1","stat":"-12","state":"FAILNOTIFY","MAC":"001A22092FB7","u":"128","svc":"3e135142-654f-9090-134a-a6ff5bb77046","char":"3fa4585a-ce4a-3bad-db4b-b8df8179ea09","notifychar":"d0e8434d-cd29-0996-af41-6c90f4e0eb2a","write":"03180A130F2C09000000000000000000"}}
15:44:15.956 BLE: failed subscribe for notify
It works with the Calorbt app. 5 other EQ3 have responded.... Tas version 14.3.0.1 on esp32S3
I'll check it out some more.
@Staars Cristian - help :). You have latest and intimate knowledge of esp-nimble-cpp.... @h2zero too if you are able.
We're hitting AddLog(LOG_LEVEL_ERROR,PSTR("BLE: failed subscribe for notify")); - line 1921 in tasmota\tasmota_xdrv_driver\xdrv_79_esp32_ble.ino
This is happening only for EQ3 firmware versions which have 'new' security. Seems to me that the device is connected, has a valid characteristic, says that characteristic can notify, but then nimble is telling us it can't subscribe.... (pNCharacteristic->subscribe returns false).
Any advice or guidance appreciated. br, Simon
In line 1911:
if(pNCharacteristic->subscribe(true, BLE_ESP32::BLEGenNotifyCB)) {
You try to subscribe to an indication and not a notification. Is this intended?
Ahh, no ... it is the opposite. forget it.
But at least in 1911 an incomplete function signature is used - missing the response
. IDK if this is a problem.
14.1.0 (lvgl-haspmota) - not working, trying 14.0.0.2..... not working. 13.4.1.1 (lvgl-haspmota)- BLE does not work at all. 13.4.0(lvgl-haspmota) none of the eq3 work. maybe something is wrong with my build.
14.1.0 (lvgl-haspmota) - not working, trying 14.0.0.2..... not working. 13.4.1.1 (lvgl-haspmota)- BLE does not work at all. 13.4.0(lvgl-haspmota) none of the eq3 work. maybe something is wrong with my build.
See the opening post of this thread. I linked the build for 13.4.0 there. Just download that one. It won't have yourWifi or mqtt settings of course, but should at least work for testing.
If I was to guess I would say that it's failing to subscribe because it should write to the descriptor with response request as per BT specifications. The older devices/firmwares wouldn't accept that in the past so that's the reason for writing without response now. At least that's what i remember.
come to it in the morning, and 13.4.0 on S3 is working for all Eq3, including ones running older firmware and the one running 1.46 - it must have just been taking it's time getting the data, and I rushed the assessment.
I'll walk forward again in versions.
@Staars - response
is defaulted to true in the header for the function. but from h2zero, maybe this is the issue.... maybe it needs false.
@h2zero - could this https://github.com/h2zero/esp-nimble-cpp/commit/5b24c8d681f967be85dd6dd1566657f7a47b0c06 be involved? - or is that only for BLE servers?
edit ... @Staars - can we turn on nimble logging in tas? if so exactly how? - I have console on this device.
Edit2: on version: 14.0.0.1 (bluetooth) 3_0_0/5.1.3.240430 it's working for old Eq3 firmware, but not new. This is just after we replaced nimble_arduino with esp-nimble-cpp - commit of my scanning fixes 2eccc96c62e1d169130266e726a506961c4b1d67
So the issue is related to our change to esp-nimble-cpp, which means we're not going to find a code change that 'caused' the issue. I'll walk all the way forward to latest dev, and try to diagnose and fix the issue there.
@Staars @h2zero - the PR works with both of the versions of EQ3 firmware that I have.
Please have a quick look and comment.
(I thought this may be in the characteristics properties, but the props are always characteristic props 0x1A
for both firmwares).
@AleXSR700 - I can't seem to find individual artifacts for the PR, but I assume the firmwares are all in https://github.com/arendst/Tasmota/actions/runs/11424841639/artifacts/2078856871 - 138mbytes.
@Staars @h2zero - the PR works with both of the versions of EQ3 firmware that I have. Please have a quick look and comment. (I thought this may be in the characteristics properties, but the props are always
characteristic props 0x1A
for both firmwares).
I suppose the needed response value is device specific (here for the EQ-3). So the PR looks good to me.
Since a few days we can easily customise the Arduino framework for a specific Tasmota build, which is especially useful for (before impossible) individual Bluetooth settings including the log level.
If it is not enough to add a build flag, like: -DCONFIG_NIMBLE_CPP_LOG_LEVEL=3
to build_flags
you can bake these kind of things into the Arduino framework now with:
custom_sdkconfig = CONFIG_NIMBLE_CPP_LOG_LEVEL=3
I did not test this particular use case yet. Maybe you need to provide both sections, it is still a very new feature and not finally finished.
@Staars - did you test notify to get battery from MI in your new implementation yet?
The PR mod will affect MI32 when using BLE32 - i.e. it uses the same notify fn.
If you test and find that response
MUST be true for the MI battery read fn, then I'll make it EQ3 specific somehow.... - let me know.
(I don't have an MI style device that required connection....)
Thank you all! I was away for the weekend but will also test tomorrow :)
@Staars - did you test notify to get battery from MI in your new implementation yet? The PR mod will affect MI32 when using BLE32 - i.e. it uses the same notify fn. If you test and find that
response
MUST be true for the MI battery read fn, then I'll make it EQ3 specific somehow.... - let me know. (I don't have an MI style device that required connection....)
As I converted basically all my devices to PVVX I can not tell ATM. It was a cumbersome procedure anyway and I suggest to wait for "real world" user issues. But if I stumble across findings I will report here.
I am not sure if this is possible, but the ble experts here might be able to say: Is it possible that the EQ3 driver makes the devices with mixed wifi-ble chip less stable?
I have 8 Mijia temp+hum sensors and 8 EQ3s. My esp32 devices (esp32 & esp32c3) work without crashes when I only use the Mijia devices (e.g. in the versions I built without EQ3 support or when the driver was broken).
With EQ3 is seems that when the TRVPeriod is lowered, it increases the crash frequency. So with TRVPeriod 300 my esp32c3 devices crash multiple times per day. When I increase to 3600, the devices seem to not crash more than once.
It is still too early since the update to say for sure, but is it possible that somehow the EQ3 driver blocks the wifi-ble module too long or something like that? Layman talking here...
the versions I built without EQ3 support or when the driver was broken
was it like this?:
#define USE_MI_ESP32
#define USE_BLE_ESP32
//#define USE_EQ3_ESP32
i.e. using the BLE_ESP32 version of BLE?
There is a big interaction between BLE of any form and wifi. Basically on ESP32, you CAN'T mix them. Really for BLE and TAS we should have one device doing BLE, and another doing wifi.
e.g. when I built webcam+ble, and then used the webcam UI, no device would last more than ~30 mins, some crashed in < 1 min.
Reducing TRVPeriod -> more MQTT & more time with active BLE connections -> more BLE + more wifi -> more crashes :(.
Using just MI sensors, it's using very little BLE, and so the risk is reduced (basically just hearing adverts, no active connections).
see also https://github.com/arendst/Tasmota/issues/20970 and other reported BLE issues.
Why is an active connection needed for EQ3 but not Mijia? The esp32 devices receive data without active connection. But they need an active connection for the query? If so, can the duration if this active connection maybe be reduced/optimized?
Again, just asking as a layman.
I have found that the active checking via the TRVPeriod is rarely needed. The TRV will report any change without being queried (and that is received by the tasmota esp32 without active connection). So only an initial query is needed to get the starting value whenever the esp32 device or the Home Automation server are restarted. Apart from that, any change will be actively broadcast by the TRV and received by the esp32 tasmota devices.
@AleXSR700 - sorry for the delay. The EQ3 needs to be read
, which requires a connection, whereas the MI devices broadcast their stats in BLE adverts - so anyone can hear the info without connecting. The exception to this is some MI devices don't include battery level, but this can be read using a connection.
So far as I remember, 'any change will be actively broadcast by the TRV' is not true. We will only get updated info from the TRVs when we ask for it. (i.e. at the TRV interval).
So far as I remember, 'any change will be actively broadcast by the TRV' is not true. We will only get updated info from the TRVs when we ask for it. (i.e. at the TRV interval).
Thanks for your response. Nope, the above is definitely true. I have set the TRVPeriod to 1 hour and I get constant updates when the TRV is running. It sends updates whenever there is a change in temperature (I think also valve). No need to ask for it :)
Just manually change the temperature at the dial or via HA and you will see that it broadcasts once it has received the command (be it via dial or via MQTT).
I can agree with btsimonh's observations. If TRVperiod is set to 0, no values come from the valves. Regardless of whether I control them via the app or change the temperature on the dial.
I change the value with the status of my heating circuit.
Heating circuit pumps on --> TRVPeriod 300 Heating circuit pumps off --> TRVPeriod 0
When the circuit is not running, I can only see the current temperature setting, but only the last valve position. But that doesn't interest me at all at this point. An active query using ‘TRV status’ is always possible.
I can agree with btsimonh's observations. If TRVperiod is set to 0, no values come from the valves. Regardless of whether I control them via the app or change the temperature on the dial.
Please set TRVPeriod to something high, like 3600 and then use the dial. You should see the publish in the console. I know I do. So either that is new in TRV firmware 1.46 or TRVPeriod 0 disables the TRV all together (have not tested that myself).
@btsimonh I thought about this a bit more and I was wondering if maybe we are staying connected "needlessly".
For sure the TRV publishes whenever there is a change and tasmota does not need to connect to receive that message. Essentially it behaves like a Mijia thermometer etc. They publish and anyone within reach can listen. Not exactly safe, but what the heck...
So I think right now you are connecting, querying the status and waiting for the response before disconnecting. I think you could simply send a command to the TRV and then wait for the response without being connected.
Any valid command will trigger the publish and any device will see it (without being connected).
That should significantly reduce the duration of the connection to the pure time that is needed for sending the command.
P.S.: Happy to be your guinea pig if you think that could be modified easily enough :)
If you command a change, e.g. via HA or other thing connected to tasmota, TAS sends the request (e.g. to change the temp), and the response contains the new state (all the values we get from the TRV come in one message, always the same, via 'notify' in response to us asking anything). If you adjust the temp on the TRV itself, I would expect the new value to only be seen by TAS after the TRVPeriod has expired, and the TRVs actively read again. However, if the demand to change the temp is sent by TAS, the new values are immediately available in the response to the request to change.
The advertisments from the TRVs (data send periodically and without connection) do not contain any useful data for us. The only way to get useful data is to connect to each TRV in turn, and make some request (any - since upon any request, the TRV will send a 'notification' containing the latest data). In order to receive a notification, you must be connected.
TAS/the low level BLE stack can have a maximum of three concurrent connections, so we only ever open one connection at a time, cycling through the TRVs once per TRVPeriod, and closing the connections once we have a response or a failure.
On failure, we do retry (3 times?).
During a 'failure mode' (e.g. TRV far away), the polling for the data may occupy many seconds, during which we don't pass on other requests (they are queued). Hence why we can set aliases, and only poll the TRVs we wish from a specific TAS - to avoid polling a 'too distant' device where this is known to be problematic.
The advertisments from the TRVs (data send periodically and without connection) do not contain any useful data for us. The only way to get useful data is to connect to each TRV in turn, and make some request (any - since upon any request, the TRV will send a 'notification' containing the latest data). In order to receive a notification, you must be connected.
Here I am not sure if this is not a contradiction to your first paragraph.
As you mentioned, when you send a command, the TRV will respond with the data we need. So you do not need to stay connected and wait for a response, you just need to send a command and then you will automatically get a response. You could of course add a verification, i.e. send the command, remain disconnected, send another command if there is no response within 5 s of the first command. Only connect if this fails 3 times.
Should this not significantly reduce the connection duration and hence the shared wifi/ble blocking?
PROBLEM DESCRIPTION
A clear and concise description of what the problem is.
I do not know if this is related to issue https://github.com/arendst/Tasmota/issues/21445, but after upgrading my esp32/esp32c3 devices to 14.2.0.6, my eQ3 devices are no longer connecting reliably.
I have 8 Xiaomi devices and 8 EQ3s. The Xiaomis (running ATC firmware) seem to work but the EQ3s just show
20:44:07.951 EQ3: 001A2208C9FA: trv operation failed - retrying -12
(with the respective device ID).Interestingly, I see all 8 EQ3 TRVs right after a restart and they briefly show the correct temperatures in the WebUI. But then the connection starts failing and all values are set to 0.
Does anybody have similar issues?
EDIT: Reverted to 13.4.0 (http://ota.tasmota.com/tasmota32/release-13.4.0/tasmota32-bluetooth.bin) and it works again. So seems that something is broken in 14.2.0.6 again.
REQUESTED INFORMATION
Make sure your have performed every step and checked the applicable boxes before submitting your issue. Thank you!
Backlog Template; Module; GPIO 255
:Status 0
:TO REPRODUCE
Steps to reproduce the behavior:
Upgrade to 14.2.0 and enable BLE (if not already enabled). Check console for EQ3-TRV messages.
EXPECTED BEHAVIOUR
A clear and concise description of what you expected to happen.
SCREENSHOTS
If applicable, add screenshots to help explain your problem.
ADDITIONAL CONTEXT
Add any other context about the problem here.
(Please, remember to close the issue when the problem has been addressed)