Descriptions for undocumented characteristics

[ariccio]

As part of work on an app I run, I did some spelunking. I have figured out the purposes of the characteristics that are not yet documented:

• "f0cd1401-95da-4f4b-9ac8-aa55d312af0c" is the sensor settings state, packed into a small struct.
• "f0cd1502-95da-4f4b-9ac8-aa55d312af0c" is the sensor calibration.
• "f0cd2003-95da-4f4b-9ac8-aa55d312af0c" appears to be unused - no reference to it in the entire app. Maybe that's why it contains all zeros?
• "F0CD2005-95DA-4F4B-9AC8-AA55D312AF0C" appears to be the characteristic for the sensor logs. I haven't written out the data layouts yet or wrote code to parse them, but thought I'd let you know!

[ariccio]

Ok, it also looks like the "XX" unknown field in the current readings GATT is actually a bitfield that describes the color of the current reading. If bit 1 is set, it's either red or yellow (by bit zero). Here, bit zero set means red, and not set is yellow.

If bit 1 is not set, it's either green, or unspecified (by bit zero). Here, bit zero set means green, not set is unspecified.

[0ki]

so, in decimal: 0 - unknown 1 - green 2 - yellow 3 - red

;)

[ariccio]

LMAO yes, maybe it's just an enum! You should see how they're parsing it in the aranet4 app... I'm tempted to post in their forum about it, but I don't want them to get worried about people reverse engineering their code and then start obfuscating it. They shouldn't, after all, it's a super simple react native app, and if people know how to use the protocol, of course, then more people might buy the device!

[Anrijs]

Thanks! Docs have been updated. F0CD2005-95DA-4F4B-9AC8-AA55D312AF0C could be new characteristic, added in later firmware. I will try to take a look at it.

[ariccio]

Ok, why not? So the funny thing about the "color" field, again is the way they decided to parse it. If it was intended to just be 1,2,3, why the heck would they do this?

        O = function(t) {
            return ('0'.repeat(8) + t.toString(2)).slice(-8).split('').map(parseFloat).reverse()
        },

Honestly, it is the most unintentionally obfuscated way to parse a bitfield that I've ever seen :D

[sbinet]

do you know what is the meaning of the data one gets back when passing a param==5 value to pullHistory ? (after a bit of tinkering param==5 is the last value for which the device doesn't error back out)

[ariccio]

Which one is pullHistory? I can check my notes.

[sbinet]

this: https://github.com/Anrijs/Aranet4-Python/blob/9e12cb75960c7c837aa3328512a09fe61a6cf46f/aranet4/client.py#L262

when I was testing my Go driver (here), I noticed I could pass 1, 2, 3, 4 and 5 as command parameter (and get back something weird with 5)

[ariccio]

Ah, you're talking about what @Anrijs calls the "Set history parameter"? If you pass 6, what happens? I'm not quite good enough of a reverse engineer yet with the metro packer, but it looks like 0 and 5 are special, and refer to variables.

There appear to be some bits in the calibration code that write "ts" somewhere? That characteristic seems to handle a lot of arbitrary things, like setting bluetooth range, Homey smart home integration, setting the buzzer, setting the thresholds for warning levels, etc...

[ariccio]

And no, the firmware image for the device is encrypted, I've gone down that route. And I'm not yet going to rip mine apart to dump it.

[sbinet]

with param > 5, one gets a 0xee error code.

[elyscape]

I've figured how to retrieve and interpret the history data using f0cd2005-95da-4f4b-9ac8-aa55d312af0c.

First, you write a 4 byte value with the following layout to f0cd1402-95da-4f4b-9ac8-aa55d312af0c: 61:TT:SS:SS

Field	Value	Type
TT	Type of measurement to retrieve	u8
SS:SS	First index to return	uLE16

Then you repeatedly read f0cd2005-95da-4f4b-9ac8-aa55d312af0c. The value read will have a 10-byte header and then a payload of measurements. The header looks as follows: TT:II:II:RR:RR:UU:UU:SS:SS:LL

Field	Value	Type
TT	Measurement type	u8
II:II	Measurement interval (seconds)	uLE16
RR:RR	Total measurements stored on device	uLE16
UU:UU	Time since last measurement (seconds)	uLE16
SS:SS	Index of first measurement in payload	uLE16
LL	Number of measurements in payload	u8

When uLE16(SS:SS) + u8(LL) - 1 == uLE16(RR:RR), you have read all the data requested.

Some notes:

• As with the previous history mechanism, the measurement index starts at 1.
• Measurements in the payload are in chronological order.
• The length of the payload will always be either 0 bytes or 234 bytes, irrespective of the value of LL. All trailing bytes are garbage and should be discarded.
• If the device is completing a measurement at the moment the value is read, a few things happen:
  • TT will be 129 and LL will be 0. There will be a payload, but it is garbage and should be discarded.
  • The device will decrement SS:SS by 1 and return an additional data point. Which is to say, it returns all the requested historical measurements and also the new one that it was taking at the moment of retrieval.

[KASSIMSAMJI]

I've figured how to retrieve and interpret the history data using f0cd2005-95da-4f4b-9ac8-aa55d312af0c.

First, you write a 4 byte value with the following layout to f0cd1402-95da-4f4b-9ac8-aa55d312af0c: 61:TT:SS:SS Field Value Type TT Type of measurement to retrieve u8 SS:SS First index to return uLE16

Then you repeatedly read f0cd2005-95da-4f4b-9ac8-aa55d312af0c. The value read will have a 10-byte header and then a payload of measurements. The header looks as follows: TT:II:II:RR:RR:UU:UU:SS:SS:LL Field Value Type TT Measurement type u8 II:II Measurement interval (seconds) uLE16 RR:RR Total measurements stored on device uLE16 UU:UU Time since last measurement (seconds) uLE16 SS:SS Index of first measurement in payload uLE16 LL Number of measurements in payload u8

When uLE16(SS:SS) + u8(LL) - 1 == uLE16(RR:RR), you have read all the data requested.

Some notes:

* As with the previous history mechanism, the measurement index starts at 1.

* Measurements in the payload are in chronological order.

* The length of the payload will always be either 0 bytes or 234 bytes, irrespective of the value of `LL`. All trailing bytes are garbage and should be discarded.

* If the device is completing a measurement at the moment the value is read, a few things happen:

  * `TT` will be 129 and `LL` will be 0. There will be a payload, but it is garbage and should be discarded.
  * The device will decrement `SS:SS` by 1 and return an additional data point. Which is to say, it returns all the requested historical measurements and also the new one that it was taking at the moment of retrieval.

Hi Elyscape, I am trying to implement what you have explained here but things get over my head

Here is my reference link

FYI my ESP32 acts as Aranet4 device, sending data to the Aranet4 Home

On the app when I hit the graph icon, I see this on Arduino's serial monitor

f0cd1402-95da-4f4b-9ac8-aa55d312af0c: onWrite(), value: 0x61,0x04,0x00,0x00

That means the app is asking for data history logs beloging to CO2 from the device, to which I reply with

 struct data_struct {

    // start 10 Bytes header

    uint8_t measure_type = 0x04
    uint16_t measurement_interval_sec = 10;
    uint16_t total_measurement_stored = 30;
    uint16_t time_since_last_measure_sec = 5;
    uint16_t index_of_first_measurement_in_payload = 1;    
    uint8_t number_of_measurement_in_payload = 10;          

    // index_of_first_measurement_in_payload + number_of_measurement_in_payload  == total_measurement_stored, you have read all the data requested.

    // end header

    // start payload

    uint16_t a1 = 12;    // humidity needs uint8_t
    uint16_t b1 = 13;
    uint16_t c1 = 14;
    uint16_t d1 = 15;
    uint16_t e1 = 16;
    uint16_t f1 =  17;
    uint16_t g1 = 18;
    uint16_t h1 = 19;
    uint16_t i1 =  20;
    uint16_t j1 = 21;
    uint16_t k1 = 22;
    uint16_t l1 = 23;
    uint16_t m1 = 24;
    uint16_t n1 = 25;
    uint16_t p1 = 26;

    // end payload
  }  __attribute__((packed)) data_to_send;

The app shows a prompt "Loading 0%..."

Then I keep receiving back f0cd1402-95da-4f4b-9ac8-aa55d312af0c: onWrite(), value: 0x61,0x04,0x00,0x00 repeatedly , I have no idea how I should re-structure my response this time

Could you kindly correct my data structure in the response above ?

and also correct it on how It should look like each time I receive f0cd1402-95da-4f4b-9ac8-aa55d312af0c: onWrite(), value: 0x61,0x04,0x00,0x00 , that would save me ton of hours

Much Thanks in advance

[elyscape]

That means the app is asking for data history logs beloging to CO2 from the device, to which I reply

This is your mistake. The response should be empty, and the data you're sending back in the response is actually read from the f0cd2005-95da-4f4b-9ac8-aa55d312af0c characteristic. The flow looks like this:

1. App writes 61:04:00:00 to f0cd1402-95da-4f4b-9ac8-aa55d312af0c. This sets the device's history retrieval state to be CO2 at index 0.
2. App reads f0cd2005-95da-4f4b-9ac8-aa55d312af0c. Device responds with the data_to_send struct you described and increments the index of its retrieval history state by number_of_measurement_in_payload.
3. App repeats step 2 while (index_of_first_measurement_in_payload + number_of_measurement_in_payload - 1) < total_measurement_stored.

[KASSIMSAMJI]

That means the app is asking for data history logs beloging to CO2 from the device, to which I reply

This is your mistake. The response should be empty, and the data you're sending back in the response is actually read from the f0cd2005-95da-4f4b-9ac8-aa55d312af0c characteristic. The flow looks like this:

1. App writes `61:04:00:00` to `f0cd1402-95da-4f4b-9ac8-aa55d312af0c`. This sets the device's history retrieval state to be CO2 at index 0.

2. App reads `f0cd2005-95da-4f4b-9ac8-aa55d312af0c`. Device responds with the `data_to_send` struct you described and increments the index of its retrieval history state by `number_of_measurement_in_payload`.

3. App repeats step 2 while `(index_of_first_measurement_in_payload + number_of_measurement_in_payload - 1) < total_measurement_stored`.

Hi Eli, Thansk for your input

I am seeing a little success here

here is my code snippet so far

void sending_data()  {

  struct data_struct {

    // start 10 Bytes header

    uint8_t measure_type = measurement_type;  // 0x04 for CO2
    uint16_t measurement_interval_sec = 10;
    uint16_t total_measurement_stored = 10;
    uint16_t time_since_last_measure_sec = 5;
    uint16_t index_of_first_measurement_in_payload = 1;    // this one increment
    uint8_t number_of_measurement_in_payload_2 = number_of_measurement_in_payload;           // this finaly goes to 0

    // start payload

    uint16_t a1 = random(400, 999);    
    uint16_t b1 = random(400, 999);
    uint16_t c1 = random(400, 999);
    uint16_t d1 = random(400, 999);
    uint16_t e1 = random(400, 999);
    uint16_t f1 = random(400, 999);
    uint16_t g1 = random(400, 999);
    uint16_t h1 = random(400, 999);
    uint16_t i1 = random(400, 999);
    uint16_t j1 = random(400, 999);

    // end payload
  }  __attribute__((packed)) data_to_send;

  // device_2data.co2 = random(400, 999);

  std::string formatted_data((char *)&data_to_send, 10 + (2 * number_of_measurement_in_payload));  // NOTE THIS LINE HERE PLEASE

  pSensorLogsCharacteristic->setValue(formatted_data);

}  // end function

as soon as I am seeing f0cd1402-95da-4f4b-9ac8-aa55d312af0c: onWrite(), value: 0x61,0x04,0x00,0x00,

I set number_of_measurement_in_payload = 1 then I invoke sending_data() to write the struct data to f0cd2005-95da-4f4b-9ac8-aa55d312af0c

whenever the app reads from f0cd2005-95da-4f4b-9ac8-aa55d312af0c I increment number_of_measurement_in_payload++ then sending_data() is invoked again, and again

Here is what I am seeing on the app, "Loading Carbondioxide 10%" , "Loading Carbondioxide 20%" "Loading Carbondioxide 30%"

Then It gets all the way past "Loading Carbondioxide 100%" going upto "Loading Carbondioxide 1620%" and gets stuck there

the app keeps reading from f0cd2005-95da-4f4b-9ac8-aa55d312af0c indefinitely, and I keep incrementing number_of_measurement_in_payload++

I thought It is supposed to stop at 100% as soon as index_of_first_measurement_in_payload + number_of_measurement_in_payload - 1) < total_measurement_stored ?

Thank You Again

[elyscape]

My guess is that there's an off-by-one error. Try setting index_of_first_measurement_in_payload to 0.

[ariccio]

FYI: std::string formatted_data((char *)&data_to_send ...is a very dangerous way to serialize that data. There be dragons. I do not know the structure of your codebase, but if there's an overload for setValue that avoids such type punning, definitely not a bad idea to use it. Calculating the size manually is very dangerous too. At the very least, perhaps setValue could accept a std::byte instead? Apparently there is even a newfangled helper std::as_bytes.