How to make use of bluetooth SERVICEDATA an MANUFACTURERDATA ?

[ozett]

Is your feature request related to a problem? Please describe. OMG is sending lots of manu-data and servicedata from devices around my house. i guess, that can be decoded

dont know if i can decode this mayself in node-red or somehow like in theengs-decoder already for some devices

Describe the solution you'd like some hints, how to make use of that captured data. some hints how to start decoding the service-data and manufutererdata. it is done for some specific devices (in the source) and i would like try to add some more from my BT announcements

Describe alternatives you've considered wait that somehow a decoder is built into OMG and data is send in cleartext

Additional context captures, but what is it?

[koenvervloesem]

Welcome to the world of reverse engineering BLE data ;-)

The manufacturer data starting with 4c00 are using the Apple Continuity protocol. See https://github.com/furiousMAC/continuity for some documentation.

The manufacturer data starting with 6909 are from some Chinese company. That could need some reverse engineering. Do you have an idea which device it is? If it's a sensor, you could read its data on its display or in its app and compare with the manufacturer data.

This request is probably more something for the decoder repository.

[ozett]

i dont want to re-invent the wheel, but OMG and ESPRESENCE makes it difficult to add reverse-engeneering single devices BLE-Data. with node-red GUI it would be lots easier to fiddle and reverse-engeneer some fields of the BT-data.

ESPRESENSE makes use of the manufacturer data for fingerprinting (helps with changing BT MAC), and OMG seems that completely missing. would be nice to have with OMG

examples for company decoding on espresence: APPLE SAMSUNG: EDDYSTONE Beacons COVID-TRACKER SONOS,GARMIN,itrack,MI-FIT,Microsoft ...

examples for company decoding (and more) in theengs (?) https://github.com/theengs/decoder/tree/development/src/devices

so it could be a nice and first simple step to show/decode the manufacturer for all my BT devices wit OMG. some example for reverse-engeneering, but maybe also to tight to sourcecode, but great example

How to start with simpler decoding ?

[koenvervloesem]

Do you just want to see some more information about the manufacturer data or service data? I actually wrote a tool to help with that: HumBLE Explorer. Maybe try that for a start to explore what your devices are advertising?

[ozett]

hey, great. will look into your tool...

have lots of numbers in "manufacterdata".. maybe i find a way (reverse-eng.?) to enrich that numbers to names and values? "4c00/004c" seems lots of APPLE...

[koenvervloesem]

These numbers (4c00 for Apple and so on) are standardized. They're called the Bluetooth company IDs. There's a whole database of it.

[DigiH]

so it could be a nice and first simple step to show/decode the manufacturer for all my BT devices wit OMG. some example for reverse-engeneering, but maybe also to tight to sourcecode, but great example

How to start with simpler decoding ?

Yes, this is the description page for adding new decoders to Theengs Decoders, which OpenMQTTGateway also uses for BLE decoding.

I have created a simple Apple HomePod mini device only decoder for a while now, which you could also try and test at

https://github.com/DigiH/decoder/tree/homepod-mini

but until white/black-listing is possible by the model property we didn't really find it useful to integrate such model only decoders into the Decoder release yet, but might be planning such model decoders for the future with w/b-listing functionality.

You also have to remember that most modern devices, which also means all the Apple devices you might have, do change their MAC address about every 15 minutes for privacy reasons - unless it is being received by another Apple device which is logged into the same iCloud account - not really applicable to an ESP32 running OpenMQTTGateway though ;) plus there is also the uuid vs. MAC address issue on receiving Apple devices, when running Theengs Gateway for example.

So with your above long list it could well be that several entries there are duplicates of the same device, but with changing MAC addresses over time, and depending on how long you had MQTT Explorer running.

Apart from such awkward model/manufacturer only BLE broadcasts, if you feel you also have devices which can/should be fully decoded with useful information, please log an issue for these so we can work together on integrating them into Theengs Decoder.

[DigiH]

@ozett, are you able to build your OMG version with Platformio?

If so I could create some test model decoders for the Nearby and FindMy messages you are seeing from your Apple devices, to see how the result would fit what you are expecting.

[ozett]

@ozett, are you able to build your OMG version with Platformio?

yes, it tried it before for the das18b20 sensor and dht22, i dont really like that whole effort, and find fiddling with sourcecode is less appealing than doing "wiring" in node-red. maybe i consider this platformio a first step for the reverse-engeneering. but in node-red i could do all this with some function-node or regex and it would be open to some edits.

If so I could create some test model decoders for the Nearby and FindMy messages you are seeing from your Apple devices, to see how the result would fit what you are expecting.

i think i try this as a fist step. as a second step maybe let node-red show the "Bluetooth company IDs. There's a whole database of it. a general an more open approach could be to have an editable decoderbase. that could be done as a transfer of the teengs-decoding-logic to a node-red module. lets see how far this start here can go ..

[DigiH]

i dont really like that whole effort, and find fiddling with sourcecode is less appealing than doing "wiring" in node-red. maybe i consider this platformio a first step for the reverse-engeneering. but in node-red i could do all this with some function-node or regex and it would be open to some edits.

All that is required is to change the decoder URL from

decoder = https://github.com/theengs/decoder.git#v0.9.7

to decoder = https://github.com/DigiH/decoder#apple-nearby-findmy

in platformio.ini - rebuild and upload. No coding, no regex, no jumping through hoops … only NEARBY fully implemented so far, in case you want to try it out.

as a second step maybe let node-red show the "Bluetooth company IDs. There's a whole database of it.

The database is not so much of an issue, apart from possibly squeezing the whole databse into OMG, and therefore onto memory restricted ESP32s, when it would be only for getting a few company IDs for some undecoded devices out of the whole database., without having any further information or value becuase of the ever changing MAC addresses.

More important would also be the fact that there are many non-company ID compliant devices around in the wild. Some of them using the first two bytes for broadcasting temperature, humidity or such. How would you try and filter such devices when for example a device would send a leading 4c00 because it has a humidity of 76%, but it would then be falsely recognised as an Apple device. Then when the humidity changes over time it might go through many other companies as it's reported ID. Seemingly nice company reporting, but on closer examination just false information.

Unfortunately this is not hypothetical, but has been observed with many devices and applications using such an indiscriminate company ID database lookup.

[ozett]

trying to re-build the apple-findmy decoder with the node-red switch-node. at first sight this looks like dooable, only match on sequences of manufacterer data. (if i am not complety wrong)

why compile and build into the receiver-device? and edit, compile, and upload every time something changes? may i give it a try, but running it only to catch the data and send over MQTT looks sufficient to me..

one could add more filter like brand or existing decoded topics, but it could also easy copied to build a filter for unknown devices.

[DigiH]

I've fully implemented the NEARBY and FINDMY options now, along with some device recognition as well, for iPhone, HomePod mini and Apple TV on my side.

Just doing a CLEAN ALL, CLEAN and BUILD again with the Decoder URL staying the same will give you this currently final functionality now for testing, where your above circled 4c001006… should be decoded as NEARBY - Apple TV

[ozett]

the filter from your source and from espresence should easy be done in node-red syntax as a match to the first-data of manufacterer data with the common switch-node. right?

---

[DigiH]

the filter from your source and from espresence should easy be done in node-red syntax as a match to the first-data of manufacterer data with the common switch-node. right?

Correct, plus the additional byte following which I have partially implemented as device type property recognition in the latest decoder update.

In the end though, depending really on what you want to achieve with node-red, it might be really easier to apply a white-list to your OpenMQTTGateway, to only receive the messages from devices you want to process, instead of having to filter out all the unwanted messages.

[ozett]

a white-list would be helpful, if one could rely on the MAC. but the MAC changes nowadys randomly as default. (fingerprinting is also no option, somehow also unrelyable)

so i go on slowly to identify my house-devices and see what they are sending. than i go on slowly to see if there are existing filter which match..

[ozett]

can be done...

Node-RED flow

[ { "id": "2ab459264ff28926", "type": "mqtt in", "z": "e53ac3e112912e4f", "name": "", "topic": "omg/#", "qos": "2", "datatype": "json", "broker": "9b50c12.74a3a4", "nl": false, "rap": true, "rh": 0, "inputs": 0, "x": 90, "y": 160, "wires": [ [ "d8ef141c72ca2a94", "deea14af2a8eeef1" ] ] }, { "id": "4ff83f50b90f8ce5", "type": "debug", "z": "e53ac3e112912e4f", "name": "omg.all", "active": false, "tosidebar": true, "console": false, "tostatus": true, "complete": "true", "targetType": "full", "statusVal": "payload", "statusType": "auto", "x": 480, "y": 100, "wires": [] }, { "id": "d4274130de74fca1", "type": "switch", "z": "e53ac3e112912e4f", "name": "^4c0010=nearby", "property": "payload.manufacturerdata", "propertyType": "msg", "rules": [ { "t": "regex", "v": "^4c0010", "vt": "str", "case": false } ], "checkall": "true", "repair": false, "outputs": 1, "x": 340, "y": 180, "wires": [ [ "29deb2e4babef159" ] ] }, { "id": "29deb2e4babef159", "type": "debug", "z": "e53ac3e112912e4f", "name": "apple nearby", "active": false, "tosidebar": true, "console": false, "tostatus": true, "complete": "true", "targetType": "full", "statusVal": "msg.payload.manufacturerdata", "statusType": "jsonata", "x": 670, "y": 180, "wires": [] }, { "id": "b0ad02be022d5ebe", "type": "debug", "z": "e53ac3e112912e4f", "name": "ip", "active": false, "tosidebar": true, "console": false, "tostatus": true, "complete": "true", "targetType": "full", "statusVal": "msg.payload.ip", "statusType": "jsonata", "x": 310, "y": 100, "wires": [] }, { "id": "3005a8e77d1e777e", "type": "switch", "z": "e53ac3e112912e4f", "name": "^4c0012=findmy", "property": "payload.manufacturerdata", "propertyType": "msg", "rules": [ { "t": "regex", "v": "^4c0012", "vt": "str", "case": false } ], "checkall": "true", "repair": false, "outputs": 1, "x": 340, "y": 240, "wires": [ [ "5799033e9acfa529" ] ] }, { "id": "5799033e9acfa529", "type": "debug", "z": "e53ac3e112912e4f", "name": "apple findmy", "active": false, "tosidebar": true, "console": false, "tostatus": true, "complete": "true", "targetType": "full", "statusVal": "msg.payload.manufacturerdata", "statusType": "jsonata", "x": 670, "y": 240, "wires": [] }, { "id": "05366fba78f6bdef", "type": "switch", "z": "e53ac3e112912e4f", "name": "2=iphone,5=HomePodMini,6=A-TV", "property": "payload.manufacturerdata", "propertyType": "msg", "rules": [ { "t": "regex", "v": "^4c001[0,2]02", "vt": "str", "case": true }, { "t": "eq", "v": "^4c001[0,2]05", "vt": "str" }, { "t": "eq", "v": "^4c001[0,2]06", "vt": "str" } ], "checkall": "true", "repair": false, "outputs": 3, "x": 400, "y": 300, "wires": [ [ "9cb9346e23b0dbec" ], [ "9cb9346e23b0dbec" ], [ "9cb9346e23b0dbec" ] ] }, { "id": "9cb9346e23b0dbec", "type": "debug", "z": "e53ac3e112912e4f", "name": "apple devices", "active": true, "tosidebar": true, "console": false, "tostatus": true, "complete": "true", "targetType": "full", "statusVal": "msg.payload.manufacturerdata", "statusType": "jsonata", "x": 680, "y": 300, "wires": [] }, { "id": "d8ef141c72ca2a94", "type": "junction", "z": "e53ac3e112912e4f", "x": 200, "y": 240, "wires": [ [ "d4274130de74fca1", "3005a8e77d1e777e", "05366fba78f6bdef" ] ] }, { "id": "deea14af2a8eeef1", "type": "junction", "z": "e53ac3e112912e4f", "x": 200, "y": 100, "wires": [ [ "4ff83f50b90f8ce5", "b0ad02be022d5ebe" ] ] }, { "id": "9b50c12.74a3a4", "type": "mqtt-broker", "name": "192.168.14.204-hik", "broker": "192.168.14.204", "port": "1883", "clientid": "", "autoConnect": true, "usetls": false, "protocolVersion": "4", "keepalive": "60", "cleansession": true, "birthTopic": "", "birthQos": "0", "birthPayload": "", "birthMsg": {}, "closeTopic": "", "closePayload": "", "closeMsg": {}, "willTopic": "", "willQos": "0", "willPayload": "", "willMsg": {}, "sessionExpiry": "" } ]

[DigiH]

a white-list would be helpful, if one could rely on the MAC. but the MAC changes nowadys randomly as default.

I meant a white-list for other devices you have which do not change their MAC addresses and in which you might be interested in. From your above partial MQTT Explorer list it looks like you have several other BLE devices possible sending useful information.

Unless you are interested in these Apple device with their changing MACs, to allow for presence detection or something similar.

Unless you are in a very remote area though, and only have one iPhone, Apple Watch etc. and no close neighbours, and additionally also implement the identity resolving key for good device recognition, I can tell you that I very quickly abandoned trying to use any Apple device for such detection myself, and went for a small cheap BLE beacon on my key chain and/or a Mi Band on the arm for easy and very reliable beacon/person presence/location detection.

[ozett]

seems i found an apple device with device ID 07. @DigiH may you want to add it to your decoder?

maybe a pencil?

[DigiH]

Thanks @ozett, so from the list above you seem to have an iPhone, Apple TV, and 07, which might be the Apple Pencil. I'll add it with a current ? until we can confirm the device assignment.

[ozett]

Thanks @ozett, so from the list above you seem to have an iPhone, Apple TV, and 07, which might be the Apple Pencil. I'll add it with a current ? until we can confirm the device assignment.

can i do some other easy tests on that 07-device for name or the like?

found som e 4c00-0a and 4c00-07 types. some services other than nearby or findmy?

[DigiH]

I'm still unsure about the device IDs for the FINDMY broadcasts. Looking at my devices I think it might be the last byte instead. Will have to monitor some more.

What the 0a broadcasts indicate, and the longer 07 I don't know (yet). The 07 manufacturerdata seems even longer than an iBeacon protocol broadcast.

0a - https://github.com/furiousMAC/continuity/blob/master/messages/airplay_source.md 07 - https://github.com/furiousMAC/continuity/blob/master/messages/proximity_pairing.md

… from your linked repo above.

In the end the implementation would likely only be for recognising the services by model_id, and to be able to black-list these, as with the changing MAC addresses a clear device distinction will not be possible.

[ozett]

the air-pod thing is interesting, also battery-level and lid-open should be detecable from the soure of furiousMAC. cool

edit: also more information from nearby recoverable: https://github.com/furiousMAC/continuity/blob/master/messages/nearby_info.md now that starts to become fun...

[DigiH]

If you want to test

decoder = https://github.com/DigiH/decoder#apple-continuity

you should not see any undecoded "4c00…" messages any longer.

[ozett]

did you put decoding of the action-states and status-flags of the phones in? that would make me starting vscode .. 😃 😃 😃

https://github.com/furiousMAC/continuity/blob/master/messages/nearby_info.md

[DigiH]

No I haven't implemented them yet, ;) something to do tomorrow. Just wanted to get all the protocols done today.

The Phone or FaceTime Call interests me the most. Will have to see if it is only true during a call, or/and also when a call is coming in. The latter would be great to have incoming call/FaceTime announcements in the controller.

[ozett]

How do you approach getting the "bit-states" out of the data-string? maybe this char-encode of servicedata or manufacturer-data from the nimbleutils hexToString must be reverted?

not in your code yet (?), but looks complicated to my eyes, while vivid explained here similary for theengs-decoder

---

---

will try to decode the char-string in node-red, but i dont have a clue how to access bits in the string-values.

found some hints to convert it to buffer (one, two), but on my first tries the decoded bit-value comes from the character-value, not from the bits..

maybe i pre-calculate all values for the 3rd byte, just to see what all possible char-values that byte could become, and so have a list of possible states?

---

https://github.com/furiousMAC/continuity/blob/master/messages/nearby_info.md

---

edit: also worth decoding the battery status from 3rd byte of the findmy-data: https://github.com/furiousMAC/continuity/blob/master/messages/findmy.md

[DigiH]

How do you approach getting the "bit-states" out of the data-string?

For Decoder the bit states can be evaluated in the condition, as you posted the description above. For the direct bit assignment to desired values as a decoder you can look at the "bit_static_value" function further down in the documentation.

These two condition and decoder bit functionalities allow you to fully decode single bits or multiple bit combinations.

I have implemented the phone/facetime call functionality separately in

https://github.com/DigiH/decoder/tree/apple-call

currently with redundant model and property conditions, and it nicely works as soon as a call is coming in and the the device is ringing, as long as the call is active, and also when initiating a call.

If and how there might a differentiation in the remaining data between incoming and outgoing calls would be interesting to examine.

This also throws up the differentiation issue with multiple iPhones/iCloud accounts and related privacy issue of being able to monitor call times and durations of any such devices in the BLE receiving vicinity.

[ozett]

sounds great, the phone-decoder.

for the string-values in the manufacturer-data: at the momentet i think these are always two bytes for building the original hex-value. that must be decoded back into integer/bit values. seems dooable, even if i am a no-coder. to proof this theroy i could try to rebuild your newly phone-call decoder in node-red.

for the identification of devices from that above post https://github.com/1technophile/OpenMQTTGateway/issues/1372#issuecomment-1366808530 now i think that decoding 6th position as a device identifier is wrong, as the document from furiousmac statet that position as "length"-value of the thereafter following data.

[DigiH]

for the string-values in the manufacturer-data: at the momentet i think these are always two bytes for building the original hex-value

Often-times, but not always, as also seen in your pasted graphic above with the single-hex 4-bit Status Flags and Action Code.

now i think that decoding 6th position as a device identifier is wrong

Agreed, and already taken out yesterday ;)

[ozett]

gets harder, it the manufacterer-data string is somehow dynamic and does not follow the reverse-engeneered scheme. maybe OMG has to send raw-values to decode right? any experience with that?

https://docs.openmqttgateway.com/use/ble.html#advanced-setting-up-an-external-decoder

[DigiH]

OMG already publishes the raw manufactererdata if no decoder has been matched (https://docs.openmqttgateway.com/use/ble.html#advanced-publishing-unknown-manufacturer-s-data), or can also send the raw manufactererdata additionally along with any decoded data, if so desired and set to true

https://docs.openmqttgateway.com/use/ble.html#advanced-publishing-known-manufacturer-s-data

If always no decoding on the ESP32 gateway is wanted, but through some external decoder, your above link is the way to go, with Theengs Gateway nicely tying into this, to be able to receive undecoded raw data from one or several ESP32 getaways to then be decoded by a central Theengs Gateway.

In general the whole BLE gateway usage documentation is recommended for any OMG BLE gateway user to find out about all the possible options for get the most out of the getaways.

[ozett]

OMG already publishes the raw manufactererdata

but that raw-data seems to be encoded as chars (not numbers, or numbers as chars) in a JSON-object. so its not binary data anymore (or hex) like originally received with the esp32 and the NimBLEUtisl library. https://github.com/h2zero/NimBLE-Arduino/blob/0d9f039ba7f960946a8c9aeb639c3a64294f0d60/src/NimBLEUtils.cpp#L388 i cannot see how the received broadcast-values of, for example an int of 250 is encoded by the utils and what it looks like in the json-char-object. my coding skills are too limited to reverse out of the json the chars to hex/bitmasks again .. 😢 maybe i try tomorrow again

[ozett]

Often-times, but not always, as also seen in your pasted graphic above with the single-hex 4-bit Status Flags and Action

to my limit understanding the 2x4bit=8bit=byte is always send as 2-chars hex-value, an int250 is send as "fa" to test: http://tpcg.io/hyqz43

this must be the encoder from the bt-utils https://github.com/h2zero/NimBLE-Arduino/blob/0d9f039ba7f960946a8c9aeb639c3a64294f0d60/src/NimBLEUtils.cpp#L402

that would make it easier to decode the manu-data from the chars back to hex/binary again

[ozett]

i went over the message-block of the nearby-info and assume now, that the 4-bit action-nibble is originally encoded for the manu-data-string as a single character. also it is not stated in the naerby-document at a bitmask, what makes it easier to compare only for the listed numeric-values as the given (phone)-states.

means: substring-comparison of the 9th character is all it takes

edit: much to easy. i am totally in doubt, if that are really the correct bytes from the broadcast. how do you tested it? calling your phone?

[{"id":"cb6e67ffecaaca0b","type":"change","z":"e53ac3e112912e4f","name":"9","rules":[{"t":"set","p":"substring0","pt":"msg","to":"$substring(msg.payload.manufacturerdata, 9,1)\t","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":1150,"y":960,"wires":[["e2283f88dae27ce9","5c8acb55c3061abf"]]},{"id":"e2283f88dae27ce9","type":"debug","z":"e53ac3e112912e4f","name":"action-4bit","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"substring0","statusType":"msg","x":1310,"y":960,"wires":[]},{"id":"5c8acb55c3061abf","type":"switch","z":"e53ac3e112912e4f","name":"","property":"substring0","propertyType":"msg","rules":[{"t":"eq","v":"1","vt":"str"},{"t":"eq","v":"3","vt":"str"},{"t":"eq","v":"5","vt":"str"},{"t":"eq","v":"7","vt":"str"},{"t":"eq","v":"9","vt":"str"},{"t":"eq","v":"a","vt":"str"},{"t":"eq","v":"b","vt":"str"},{"t":"eq","v":"d","vt":"str"},{"t":"eq","v":"e","vt":"str"}],"checkall":"true","repair":false,"outputs":9,"x":1610,"y":960,"wires":[["8d431035814e771d"],["356fd9aa73f63808"],["9ba866bc46a3c635"],["54de81cdb992147d"],["3ec6d3b9d8a425de"],["2adc35bec58b020f"],["f01cb9da6d958862"],["6b891815d661803c"],["50c19d3917b68971"]]},{"id":"8d431035814e771d","type":"debug","z":"e53ac3e112912e4f","name":"action-0x01=Activity Reporting Disabled","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"$substring(payload.manufacturerdata, 9, 1) & \" (\" & $now('[Y0001]-[M01]-[D01]') & \", \" & $now('[H01]:[m01]:[s01]', '+0100') &\")\"","statusType":"jsonata","x":1900,"y":880,"wires":[]},{"id":"356fd9aa73f63808","type":"debug","z":"e53ac3e112912e4f","name":"action-0x03=Idle User","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"$substring(payload.manufacturerdata, 9, 1) & \" (\" & $now('[Y0001]-[M01]-[D01]') & \", \" & $now('[H01]:[m01]:[s01]', '+0100') &\")\"","statusType":"jsonata","x":1840,"y":940,"wires":[]},{"id":"9ba866bc46a3c635","type":"debug","z":"e53ac3e112912e4f","name":"action-0x05=Audio playing while screen locked","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"$substring(payload.manufacturerdata, 9, 1)","statusType":"jsonata","x":1920,"y":1000,"wires":[]},{"id":"54de81cdb992147d","type":"debug","z":"e53ac3e112912e4f","name":"action-0x07=Active user (screen on)","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"$substring(payload.manufacturerdata, 9, 1)","statusType":"jsonata","x":1890,"y":1060,"wires":[]},{"id":"3ec6d3b9d8a425de","type":"debug","z":"e53ac3e112912e4f","name":"action-0x09=Screen on with video playing","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"$substring(payload.manufacturerdata, 9, 1)","statusType":"jsonata","x":1900,"y":1140,"wires":[]},{"id":"2adc35bec58b020f","type":"debug","z":"e53ac3e112912e4f","name":"action-0x0a=Watch on wrist and unlocked","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"$substring(payload.manufacturerdata, 9, 1)","statusType":"jsonata","x":1900,"y":1200,"wires":[]},{"id":"f01cb9da6d958862","type":"debug","z":"e53ac3e112912e4f","name":"action-0x0b=Recent user interaction","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"$substring(payload.manufacturerdata, 9, 1)","statusType":"jsonata","x":1890,"y":1260,"wires":[]},{"id":"6b891815d661803c","type":"debug","z":"e53ac3e112912e4f","name":"action-0x0d=User is driving a vehicle","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"$substring(payload.manufacturerdata, 9, 1)","statusType":"jsonata","x":1890,"y":1320,"wires":[]},{"id":"50c19d3917b68971","type":"debug","z":"e53ac3e112912e4f","name":"action-0x0e=Phone or Facetime Call","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"$substring(payload.manufacturerdata, 9, 1)","statusType":"jsonata","x":1890,"y":1400,"wires":[]}]

[DigiH]

means: substring-comparison of the 9th character is all it takes

Exactly how I stated it above and used in my apple-call branch :)

how do you tested it? calling your phone?

Yes, including the above decoder branch in a continuous BLE OpenMQTTGateway build, it immediately triggers as soon as the iPhone rings and is active until hanging up the call.

[ozett]

this is all great news. i am getting all status now from a bunch of devices, even "driving" status at home. could be devices driving by my house, maybe. i have to test the phone-call-status with one of my iphones....

another thing: by incident i wanted to add a card to the apple-wallet and discovered, that you can make a pass on your own which is an ibeacon. this seems another cool way to make some devices unique with a .pkpass file. wondering why nobody at espresence-community discovered this yet.

read also something else, so maybe this works in another way: a pass is coming up, if the phone reads an ibeacon? https://subscription.packtpub.com/book/application-development/9781784397128/5/ch05lvl1sec36/passbook-integration

make .pkpass:

https://www.passsource.com/info/demo.php or https://tranzer.com/blogs/how-to-create-your-own-wallet-passes-pkpass/

[ozett]

is this a call? manufacturerdata: "4c0010077e1fdad6b09618"

think i now also have to compile your decoder .. 😄

[DigiH]

is this a call? manufacturerdata: "4c0010077e1fdad6b09618"

Looks like a call to me, yes

[ozett]

i am still in doubt, if the meaning of data-fields in the nearby-structure is still valid. maybe apple changed nothing, but one half of the action-byte has different meanings: https://github.com/furiousMAC/continuity/issues/7

as the manufacter-id is reversed (endiness or something?) in the original BT-Advertisment, maybe some fields must also be reversed for their original meaning? i am puzzled, as the type 0x10 shows up right, but maybe thats the length?

all the papers deciphered max IOS12, and now IOS16 is out. most wanted is a good internet source with updates from an apple-insider 😄

[DigiH]

I'm afraid I don't fully follow your concerns and confusions.

Yes, the company ID 004c is with reversed endianness leading the manufacturerdata in OMG as are all the other two octet company IDs, but all the following octets - and best to make a differentiation here between the ambiguous byte (which can be a 4-bit byte or 8-bit byte, or 16, 32 …) and octets - are in the correct order as sent out by the devices, as being published my OMG and as being used by Theengs Decoder to decode the encoded information.

Apart from some possible slightly confusing type naming conventions all the info on the Continuity Protocol pages do make perfect sense to me, at least for the parts I was looking at and playing with, while always keeping in mind that this is undocumented reverse engineered info, which might have changed in parts since their release.

Without any distinct device identification though I don't see much value in pursuing decoding and integrating any of the available data, save for being able to identify and eventually blacklist any of these Continuity broadcasts to reduce OMG MQTT noise.

Would you mind sharing what your goal is to achieve with these continuity message decoding?

[ozett]

Would you mind sharing what your goal is to achieve with these continuity message decoding?

of course, with pleasure.

i have a bunch of apple-devices in the house and wanted to see if i can identify some of them by their broadcasted activity. first problem was clarification of the data-structure, than the normal fingerprinting difficulties appear.

asuuming that all devices are not phones, only some of them, i observe confusingly lots of indications ("e") for active phone-calls. assuming my evaluation of the nearby-structure is correct, i was confused how that could happen. maybe wrong meaning of the data-field, or that are some apple-devices simply driving by my house?

i thought of using theengs-decoder, but that decoding is done the same way i build it with node-red. look for substrings and compare with the meanings of the furious-mac data-structure documents.

to investigate these possibly to many "phone-calls", i thought about a differential diagnosis of device-states. i considerd to sort out apple-watches or pencils, i considered to sort out devices with activities that are likely not to happen at the same time with i.e. a phonecall. to identify these states by the broadcast-messages, and sort out some devices, thats how my difficulties start,

every holiday-morning i tried another simple idea...without great success right now. in the search for some solutions i read the original papers,.. but i still have no other good idea (today)...

as a side-effect i sorted out all other bt-broadcasting devices in the house. there is a heating-pump boradcasting bt-data, and also i still try to make sense of parts of this data-structure. there is also garmin watch and some still unidentified devices with manufacturer-id "6909" and "e000"

[DigiH]

Thanks for the clarification, and yes, I tested the same possibilities of trying to identify my Apple devices and their states for possible integration into my home automation controller. With it currently no being possible to get clear fingerprinting though I did put the endeavour aside for the time being, only at one stage being able to tell my neighbour exactly when his last phone call on his iPhone started and how long it lasted down to the second ;)

there is a heating-pump boradcasting bt-data, and also i still try to make sense of parts of this data-structure. there is also garmin watch and some still unidentified devices with manufacturer-id "6909" and "e000"

Unknown company identifiers can be looked up on the officail Bluetooth SIG pages, but you always have to keep in mind that not all companies do apply and register for their own unique company identifier and so the assumed company ID might be misleading or in some cases actually two octets containing ever changing encoded device information. Theengs Decoder has a separate key for this.

If you feel that some of your other BLE braodcasting devices might benefit from an integration into Theengs Decoder free to open a discussion on the Decoder pages for possible decoding and integration.

[ozett]

thanks for reply, that will encourage me to prepare an esp32 with OMG and your decoder. i plan to use this second OMG device to try the MQTT-command/settings sending raw-data. will see if it gives some oportunities for substring handling of all BT-senders within node-red. will come back with this...

[ozett]

@DigiH

decoder = https://github.com/DigiH/decoder#apple-nearby-findmy https://github.com/1technophile/OpenMQTTGateway/issues/1372#issuecomment-1366010755

edit: your suggested change leads to an error... edit2: new name? -> decoder = https://github.com/DigiH/decoder#apple-continuity

---

How do i set this decoder only for my special environment? (started with fresh and new build directory and source)

---

edit3 😄😄😄 :

[ozett]

i discovered that i used the config for my previous board, what environment to use for a wemos d1 mini esp32 ?

edit: maybe like this? edit: the build flags in my environment where not used.. need a hint how to make that happen

---

edit: 🥇

[ozett]

Huh, some more information about two of my devices? at least the manufacturer? now some meaning to me for this `servicedatauuid

[ozett]

another riddle solved: more google with lots of zeros at my home:

[ozett]

some fun coming up now. servicedatauuid with meaning..

[1technophile]

Closing this, feel free to reopen if needed