Unable to get sensible sensor data from Xiaomi Flower Care

[johnhmacleod]

Thanks for this library - a huge step forwards in making BLE usable on the ESP32 - THANKS!!

I've created a sample based on your Client example to retrieve data from the Xiaomi Flower Care device. I can get the battery level & firmware info just fine but the characteristic that should return sensor data (moisture, light, fertility) is returning junk though the amount of data returned (16 bytes) is sensible. There are multiple simple Python & JS examples around to get data from this device which I have followed carefully to no avail.

http://forum.espruino.com/conversations/303598/ https://wiki.hackerspace.pl/projects:xiaomi-flora

Sample data (depending on sensor readings) should look something like "207, 0, 0, 209, 12, 0, 0, 24, 124, 1, 2, 60, 0, 251, 52, 155" - but I'm getting "170 187 204 221 238 255 153 136 119 102 0 0 0 0 0 0" - not even close!

f7 | 00 | 10 | 13 | 00 | 00 | 00 | 25 | f5 | 04 | 02 | 1c | 40 | fb | 34 | 9b
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
Temp    | ?? | Light             | %  |Fertility| ?? | ?? | ?? | ?? | ?? | ??
                                 Moisture

As I said, other data (battery, firmware) from another characteristic is fine. There are no descriptors for the characteristic used to turn on real time reporting which results in

[E][BLERemoteCharacteristic.cpp:307] retrieveDescriptors(): esp_ble_gattc_get_all_descr: ESP_GATT_NOT_FOUND

Would that be an issue?

BLE_client_ino.txt

[chegewara]

As i can understand that returned data are packed. You need to write some function that will unpack this data. try to read this code https://github.com/open-homeautomation/miflora/blob/master/miflora/miflora_poller.py#L153-L165

[johnhmacleod]

The data is simply little endian scaled integers as per the table in my post. It's definitely not correct!

So, for example:- 0xF7 0x00 at 0,1 in the table is 247 = 24.7C and 0x25 at location 7 is 37% moisture

[chegewara]

I will think about test code to see how to fix it, but in meantime. Can you read proper values with nRF connect? Also can you tell me UUIDs for service and characteristics from nRF?

PS are you working with esp-idf or arduino-ide?

[johnhmacleod]

I just manged to type fast enough in nRF to to read the values (you only get about 2 seconds before the Flower care disconnects & it printed them out in HEX for me.....

AA BB CC DD EE FF.... which is 170 187 204 221 238 255... !!!

ie The same as I get in my code. So, something else is going on. I suspect now that the write into the real time reporting characteristic is not happening for some reason, so the device is just returning default data.

UUIDs are as per my code (attached in the OP) - and the same as nRF reports.

// The remote service we wish to connect to.
static BLEUUID serviceUUID("00001204-0000-1000-8000-00805f9b34fb");

// The characteristic of the remote service we are interested in.
static BLEUUID     charUUID("00001a00-0000-1000-8000-00805f9b34fb"); // realtime reporting control
static BLEUUID    charUUID2("00001a01-0000-1000-8000-00805f9b34fb"); // Sensor data
static BLEUUID    charUUID3("00001a02-0000-1000-8000-00805f9b34fb"); // Battery & firmware

[chegewara]

In nRF connect you have macros, you can use it to write to characteristics right after connection established

[johnhmacleod]

Should clarify that it's very hard to get real data with nRF as the device disconnects very fast - well before you can write data into the control characteristic & read the sensor characteristic!

[chegewara]

i know, thats why im telling you about macros in nRF. macro will connect and send write packet for you much faster than you can do it

[johnhmacleod]

I'll check them out & get back to you.

[johnhmacleod]

OK - it returns reasonable values in nRF - 11-01-00-1A-03-00-00-00-00-00-02-3c-00-fb-34-9b That's 27.3C, 938 Lux, 0% moisture (it's on the bench!)...

And if I put it in some soil, the moisture number changes.

[chegewara]

Then now you know what to do with esp32. You need to write to characteristic right after you connect to device, right?

[johnhmacleod]

I do that! It connects & immediately finds the characteristic & writes to it.

if (connectToServer(*pServerAddress)) {
      Serial.println("We are now connected to the BLE Server.");
      connected = true;
    } else {
      Serial.println("We have failed to connect to the server; there is nothin more we will do.");
    }
    doConnect = false;
 }

  // If we are connected to a peer BLE Server, update the characteristic to enable real-time reads

  if (connected) {
      // Obtain a reference to the service we are after in the remote BLE server.
      pRemoteService = pClient->getService(serviceUUID);
      if (pRemoteService == nullptr) {
        Serial.print("Failed to find our service UUID: ");
        Serial.println(serviceUUID.toString().c_str());
        for(;;);
    }
    Serial.println("Found our service");

    // Obtain a reference to the characteristic in the service of the remote BLE server.
    pRemoteCharacteristic = pRemoteService->getCharacteristic(charUUID);
    if (pRemoteCharacteristic == nullptr) {
      Serial.print("Failed to find our characteristic UUID: ");
      Serial.println(charUUID.toString().c_str());
      for(;;);
    }
    Serial.println(" - Found our real-time reporting control characteristic");
      byte newValue[] = {0xa0, 0x1f};
      Serial.println("Setting new characteristic value");
      // Set the characteristic's value to be the array of bytes.
      pRemoteCharacteristic->writeValue(newValue, 2); // Enable real-time reporting

[johnhmacleod]

One more piece of information is that if, immediately after writing 0xA01F, I read back the characteristic, I get two bytes (as expected) but they are 0s. In nRF, they are 0xA01F.

[chegewara]

Without device or at least all logs its hard to say what is the reason. Maybe you write too fast, or read back too fast. Maybe witing procedure is wrong. Its really hard to say.

But here is the question. Does it work even when you read 0s right after writing 0xA01F? Can you read good values right after then?

[johnhmacleod]

I realise the problem of not having a device! I worked at this for some time before reaching out but ran out of ideas. I put debugging into your BLE layer but nothing came up.

Sadly, I have never seen correct data from the ESP32.

Here's the log during the write process & then the immediate read back - looks OK to me...

Setting new characteristic value
[D][BLERemoteCharacteristic.cpp:543] writeValue(): >> writeValue(), length: 2
[D][BLERemoteCharacteristic.cpp:544] writeValue(): <<    Values: 0xa0 0x1f...
[D][BLEDevice.cpp:105] gattClientEventHandler(): gattClientEventHandler [esp_gatt_if: 3] ... ESP_GATTC_WRITE_CHAR_EVT
[D][BLEUtils.cpp:1344] dumpGattClientEvent(): GATT Event: ESP_GATTC_WRITE_CHAR_EVT
[D][BLEUtils.cpp:1587] dumpGattClientEvent(): [status: ESP_GATT_OK, conn_id: 0, handle: 51 0x33]
[D][BLERemoteCharacteristic.cpp:566] writeValue(): << writeValue
[D][BLERemoteCharacteristic.cpp:432] readValue(): >> readValue(): uuid: 00001a00-0000-1000-8000-00805f9b34fb, handle: 51 0x33
[D][BLEDevice.cpp:105] gattClientEventHandler(): gattClientEventHandler [esp_gatt_if: 3] ... ESP_GATTC_READ_CHAR_EVT
[D][BLEUtils.cpp:1344] dumpGattClientEvent(): GATT Event: ESP_GATTC_READ_CHAR_EVT
[D][BLEUtils.cpp:1493] dumpGattClientEvent(): [status: ESP_GATT_OK, conn_id: 0, handle: 51 0x33, value_len: 2]
[D][BLERemoteCharacteristic.cpp:453] readValue(): << readValue(): length: 2
[D][BLERemoteCharacteristic.cpp:454] readValue(): <<    Values: 0x00 0x00 ...
The characteristic value was: 0 0 

[chegewara]

Maybe try to write 0x1f 0xa0

[johnhmacleod]

Too late, I tried that about half an hour ago!!!

Just to close the loop, I created an ESP32 server that pretends to be the Flower care - publishing the same services & characteristics (at least the ones I'm interested in). Both my ESP32 client & nRF behave properly against that server! Grrrrrr!! (By that I mean that the 0xA0 0x1F data reads back correctly after being written, then the 2nd characteristic is read correctly)

[chegewara]

Its not easy to create fake server with "strange" behaviour. I will think about it later today

[johnhmacleod]

You're right - I think it must be the server - not the client. Though why it works with nRF & trivial python code & JS code (not that I've tried the last two - but noone reports problems) I have no clue! Thanks for your responsiveness. Truly, very glad to see a usable BLE library for the ESP32. I first saw it on Andreas Speiss' YouTube channel.

[johnhmacleod]

Possibly relevant post here about doing the write & read in "one connection cycle" - same symptom. https://community.home-assistant.io/t/xiaomi-mi-plants-monitor-flower/3388/84

[johnhmacleod]

I decided to try a different tack & enable notifications for the characteristic. I find the 2902 descriptor OK but if I try writing to it, everything breaks. It doesn't seem to matter what I write. Of course, this might still be the device causing the problem!!

BLERemoteDescriptor *pRD = pRC->getDescriptor(BLEUUID((uint16_t) 0x2902));
    if (pRD == nullptr) {
      Serial.print("Failed to find our descriptor UUID: ");
      Serial.println(BLEUUID((uint16_t) 0x2902).toString().c_str());
      for(;;);
    }
    else 
      Serial.println("Got 2902");
    uint8_t data[2] = {0x00,0x00};
    pRD->writeValue(data, 2, false);
    byte newValue[] = {0xA0, 0x1F};
    Serial.println("Setting control characteristic value to 0xA01F");
    // Set the characteristic's value to be the array of bytes.
    pRemoteCharacteristic->writeValue(newValue, 2); // Enable real-time reporting

Then I get

We are now connected to the BLE Server.
Found our service
- Found our characteristics
Got 2902
Setting control characteristic value to 0xA01F
E (8137) BT: ATT - Ignore wrong response. Receives (13)                                 Request(0b) Ignored
E (8152) BT: ATT - Ignore wrong response. Receives (0b)                                 Request(40) Ignored

[chegewara]

You can write to descriptor only uint8 values in range 0-3. You have to understand this document: https://www.bluetooth.com/specifications/gatt/viewer?attributeXmlFile=org.bluetooth.descriptor.gatt.client_characteristic_configuration.xml

[chegewara]

Can you paste all code to pastebin or to gist?

[johnhmacleod]

Sure. I was going from this - says it's 16 bits... https://www.bluetooth.com/specifications/gatt/viewer?attributeXmlFile=org.bluetooth.descriptor.gatt.client_characteristic_configuration.xml

[chegewara]

Yes, its 16 bit my bad, but as you can see only bit 0 and bit 1 can be set. Other bits are reserved, has no meaning.

[johnhmacleod]

That's exactly what I was doing! Set bit 0 (only notify is supported, not indicate) and also tried clearing all bits. Same result.

[johnhmacleod]

https://pastebin.com/d4HLb71C

[johnhmacleod]

Strangely, I get different results when DEBUG is on. Still get the "wrong response" messages but the rest of the code now works (other than wrong sensor readings still!)

[chegewara]

I think i see the problem. But to be certain, do you need to write 0xa01f every time you want to read data from sensor or just one time right after you connect?

[johnhmacleod]

Probably just once as far as I know. The code actually only reads the sensors once, then loops forever. I used to have a control flag to only write 0xa01f once, but took it out as I now only read the sensors once. What do you think the issue is?

The first evidence of the difference between what I see & what nRF sees is that after writing 0xA01F, nRF is able to read that right back again from the characteristic. I only see 0x0000.

[chegewara]

Its not an issue, but try to switch notifications on descriptor 2902 with this function: https://github.com/nkolban/ESP32_BLE_Arduino/blob/f8fe9d7cdfb20caa54b70849826d1ac6e375ff78/src/BLE2902.cpp#L59

[chegewara]

Tell me which characteristic has notification and which one needs to be written with value 0xa01f? Im mean by UUID not by handler.

[johnhmacleod]

I did initially try that but I was getting crashes (probably my bad coding) so reverted to manually getting the 2902 descriptor. I'll give it another go.

Here's my code, but it crashes.

  BLE2902 *pRD = (BLE2902 *)pRC->getDescriptor(BLEUUID((uint16_t) 0x2902));
    if (pRD == nullptr) {
      Serial.print("Failed to find our descriptor UUID: ");
      Serial.println(BLEUUID((uint16_t) 0x2902).toString().c_str());
      for(;;);
    }
    else Serial.println("Got 2902");
    pRD->setNotifications(true);

Got 2902
Guru Meditation Error: Core  1 panic'ed (LoadProhibited)
. Exception was unhandled.
Register dump:
PC      : 0x400d3324  PS      : 0x00060530  A0      : 0x800d30c8  A1      : 0x3ffd3850  
A2      : 0x3ffddaf4  A3      : 0x00000001  A4      : 0xf23cf94d  A5      : 0x3ffc4268  
A6      : 0x3ffddefc  A7      : 0x00010000  A8      : 0x800d3324  A9      : 0x3ffd3830  
A10     : 0x00000000  A11     : 0x3f4016e3  A12     : 0x00000024  A13     : 0x3ffd3728  
A14     : 0x3ffc4268  A15     : 0x00000004  SAR     : 0x00000008  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000000  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xffffffff  

[chegewara]

Ok. Ive created fake device server app, and with your code its about 1150ms between you are connected to device and you write 0xa01f. This may be too long. Try to move all code which retrives and write 0xa01f to this function bool connectToServer(BLEAddress pAddress), right after this line Serial.println(" - Connected to server");

[chegewara]

This code is not needed. If connection will be established then you should have receive notifications:

   BLERemoteDescriptor *pRD = pRC->getDescriptor(BLEUUID((uint16_t) 0x2902));
    if (pRD == nullptr) {
      Serial.print("Failed to find our descriptor UUID: ");
      Serial.println(BLEUUID((uint16_t) 0x2902).toString().c_str());
      for(;;);
    }
    else Serial.println("Got 2902");
      uint8_t data[2] = {0x01,0x00};

    pRD->writeValue(data, 2, false);

[johnhmacleod]

No change I'm afraid. Here's the log with timings. It's the service/characteristic discovery that seems to takes a time.

12 - Starting Arduino BLE Client application...
420 - BLE Advertised Device found: Name: , Address: 7f:d7:9d:b9:d8:65, manufacturer data: 4c0010020b00
1025 - BLE Advertised Device found: Name: Flower care, Address: c4:7c:8d:61:ac:5f, serviceUUID: 0000fe95-0000-1000-8000-00805f9b34fb
1026 - Found our device!  address: 2029 - Forming a connection to c4:7c:8d:61:ac:5f
2029 - Created client
E (4135) BT: btc_gattc_call_handler()
3127 - Connected to server
4417 - Found our service
4419 - Setting control characteristic value to 0xA01F
E (6656) BT: No pending command
4561 - The characteristic value was: 0 0 
4561 - We are now connected to the BLE Server.
4561 - Found sensor characteristic
4681 - The characteristic value was: 170 187 204 221 238 255 153 136 119 102 0 0 0 0 0 0 
4781 - Found our battery/firmware characteristic
4806 - The characteristic value was: Battery 99% Firmware 3.1.8
E (15177) BT: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0013

[johnhmacleod]

Current code... https://pastebin.com/1eiDHYyA

[chegewara]

Im suspecting the time between those lines is too long:

3127 - Connected to server
4419 - Setting control characteristic value to 0xA01F

Can you pastebin logging from nRF connect?

[johnhmacleod]

https://pastebin.com/fKzksHZC (5 seconds!!)

[chegewara]

As i understand, from all those services, we want receive data only from 0x1204?

[johnhmacleod]

I would agree, but this JS example is so trivial.... (I'll plug in my RPi & try out the Python example)

    var currentSevice = false;
        function getData() {
          var gatt;
          NRF.connect("C4:7C:8D:62:0D:5A").then(fu­nction(g) {
            gatt = g;
            return gatt.getPrimaryService("00001204-0000-10­00-8000-00805f9b34fb");
          })
          .then(function(service) {
            currentService = service;
            return service.getCharacteristic("00001a00-0000­-1000-8000-00805f9b34fb");
         })
         .then(function(characteristic) {
             return characteristic.writeValue([0xa0, 0x1f]);
         })
         .then(() => {
             return currentService.getCharacteristic("00001a­01-0000-1000-8000-00805f9b34fb");
         })
         .then((characteristic) => {
            return characteristic.readValue();
         })
         .then(function(value) {
            console.log("temperature: ", value.getI­nt16(0, true)/10);
       console.log("Sunlight: ",value.getInt­32(3,true));
       console.log("Moister: ", value.getUin­t8(7,true));
    console.log("Fertility: ", value.getUint16(8,true));
            console.log(JSON.stringify(value.buffer)­);
           gatt.disconnect();
           console.log("Done!");
         })
           .catch(function(error){
             console.log("error", error);
           });
        }
        setWatch(function() {
          getData();
        }, BTN, { repeat:true, edge:"rising", debounce:50 });

And this Python one even more so...

#Read data from Xiaomi flower monitor, tested on firmware version 2.6.6 

from gattlib import GATTRequester, GATTResponse
from struct import *

address = "DE:AD:BE:EF:CA:FE"
requester = GATTRequester(address)
#Read battery and firmware version attribute
data=requester.read_by_handle(0x0038)[0]
battery, version = unpack('<B6s',data)
print "Battery level:",battery,"%"
print "Firmware version:",version
#Enable real-time data reading
requester.write_by_handle(0x0033, str(bytearray([0xa0, 0x1f])))
#Read plant data
data=requester.read_by_handle(0x0035)[0]
temperature, sunlight, moisture, fertility = unpack('<hxIBHxxxxxx',data)
print "Light intensity:",sunlight,"lux"
print "Temperature:",temperature/10.,"°C"
print "Soil moisture:",moisture,"%"
print "Soil fertility:",fertility,"uS/cm"

[johnhmacleod]

And the python example works perfectly!

sudo python xi.py
Battery level: 99 %
Firmware version: '3.1.8
Light intensity: 548 lux
Temperature: 27.5 C
Soil moisture: 0 %
Soil fertility: 0 uS/cm

[chegewara]

Code is really trivial and evrything looks good. Then we need to change approach. Try with this code:

static BLEUUID     charUUID((uint16_t)0x1a00);
static BLEUUID    charUUID2((uint16_t)0x1a01);
static BLEUUID    charUUID3((uint16_t)0x1a02);

[johnhmacleod]

The more things change, the more they stay the same!!

No improvement

[chegewara]

Like i said, without device i can only guessing, and i know that my code change does nothing because its equal with yours. But since the code to drive this device is so trivial, then there is something trivial that makes difference.

[johnhmacleod]

I like your reasoning. It's frustrating not knowing what's making the difference between (python & nRF) and BLE on the ESP32. Trivial != Easy unfortunately. And much harder not having a device.

Aliexpress sell them - be sure to get an international version

I suspect I may well have overstayed my welcome on this issue, so feel free to do something more useful! If I come up with anything, I will be sure to let you know. I'm also happy to try things out if you choose to persist!!

One other Q - BLE && WiFi - did I read somewhere that trying to use them together on the ESP32 is fraught with danger? I work for IBM & use the ESP8266 heaps for IoT demo projects. So my interest in BLE on the ESP32 is as an enabler for a low power devices to talk to our IoT Platform. However, if BLE+WiFi on the ESP32 is flakey (for now, at least), I might need to look elsewhere.

[chegewara]

Its not a problem, i like to struggle with such problems. Do you mind to switch to esp-idf or you preffer to stay with arduino-ide?

I think its not a problem when you have some knowledge and experience to make esp32 working BLE+WIFI. Of cource its easier to do that in esp-idf environment, but i think it should be easier when we get arduino-ide updated to esp-idf v3.0. Sure, you can find more mature device for your need, but i think esp32 firmware in version 3.0 is almost ready to use in production.

[johnhmacleod]

I can give esp-idf a go - never tried using it. [work to do]

I'll certainly give the BLE+WiFi combo a go then. Was hoping to set up the ESP32 as a gateway from the Flower care - as a PoC. Looks like I need to find another BLE sensor. I have plenty of TI CC2650 SensorTags - I'll try that, maybe.

[chegewara]

You provided me all pieces i need to solve this. Now i need to read it thorough and think hard.

[chegewara]

I have one more idea. Can you delete BLE library from arduino libraries and add this one?

ESP32_BLE.zip

I have to warn you. It will be a little bit messy, and you will need to edit few files before it starts to work. But then you can use sketch you have. But first we have to check what version of toolchain you have installed. You can find it in this file: ~/Arduino/hardware/espressif/esp32/package/package_esp32_index.template.json

[johnhmacleod]

1.22.0-75-gbaf03c2-5.2.0