userx14 / omblepy

Cli tool to read records from omron blood-pressure bluetooth-low-energy measurement instruments
44 stars 12 forks source link

Documentation #2

Closed bonnybabukachappilly closed 1 year ago

bonnybabukachappilly commented 2 years ago

Can you tell me how you find which UUIDs for different functions and bytes code to write for a particular action?

userx14 commented 2 years ago

Hi @bonnybabukachappilly ,

so the relevant UUIDs can be found here https://github.com/userx14/omblepy/blob/63fd4919bfb851d23e1bf031cbc054eb9627e2ed/omblepy.py#L24-L32

which UUIDs for different functions

As an overview for which UUID is used on what:



bytes code to write for a particular action

So far I know of the following byte sequences for the CommandChannel:

Generally all packets on the RX and TX channels, are always formatted like so:



The things you need to do to get the data are:

  1. Unlock the device by writing to "deviceUnlock_UUID" and checking the response on the same UUID. What bytes you need to write here depends if the device was already paired once or not (see writeNewPairingKey() or unlockWithPairingKey())
  2. Subscribe for notifications on the 4 RX characteristics UUIDs
  3. Send the "start transmission" command on the deviceCommandChannel_UUID (https://github.com/userx14/omblepy/blob/63fd4919bfb851d23e1bf031cbc054eb9627e2ed/omblepy.py#L105) and wait for a response to arrive at the rx channels
  4. Depending on a device, the address where the data for each user is stored at is different. Use readContinuousEepromData() to request the readout of large chunks data, which itself splits the reading in blocks to comply with size limitation of the RX, channels. To read such a block you need to write the sequence dataWriteCommand in _writeBlockEeprom() to the deviceCommandChannel_UUID characteristic
  5. After writing the characteristic you will get callbacks for the data blocks on each of the four rx channels. You need to combine the data of these channels.



The thing you probably need to change to add support for you device is to just add a file in the device specific config folder and find out from which address the data is read on your device. But it would be helpfull to know which device you are trying to add support for.

Best, Benjamin

userx14 commented 2 years ago

Ok I have just seen that you are using omron 7361t

The reason that this one is not supported yet is, that is has a different byte order in the records itself (hence the different import of bytearrayBitsToIntLittleEndian). If you can give me some time I can hopefully work it out. If you want to do some testing in the meantime you should be able to use:

import sys
import datetime
sys.path.append('..')
from utility import bytearrayBitsToIntLittleEndian

def recordSize():
    return 0x10
def startAddress():
    return 0x98
def user1Entries():
    return 100
def user2Entries():
    return 100

def recordToDict(recordBytes):
    recordDict = dict()
    print(f"{recordBytes}")
    #Well that's the reason I have no added support yet
    return recordDict 

as the device specific config file. It which will crash the code in a later stage of the programm, but should print the raw bytes of the records before that.

Maybe you could give me one or two records bytesArrays to test on.

bonnybabukachappilly commented 2 years ago

Let me try, I will get back to you.

bonnybabukachappilly commented 2 years ago

'The reason that this one is not supported yet is, that is has a different byte order in the records itself'

what records

userx14 commented 2 years ago

'The reason that this one is not supported yet is, that is has a different byte order in the records itself'

So each measurement you take is stored in the eeprom with the time of recording, the sys pressure, the dia pressure, and the heart rate and some other stuff. Omron calls each such stored measurement a "record", so I have copied their nomenclature here. Presumabelly to save eeprom space and cut down the transmission time, Omron has decided to compress the records were they can. For my device this results in: Capture And that are the values you see hardcoded in https://github.com/userx14/omblepy/blob/63fd4919bfb851d23e1bf031cbc054eb9627e2ed/deviceSpecific/hem-7322t-d.py#L17-L27 And in the code of UBMP this corresponds to: https://codeberg.org/LazyT/ubpm/src/commit/815a74dce1c41b16594e9e2d6208f96f3f2999a2/sources/plugins/vendor/omron/hem-7361t/DialogImport.cpp#L442-L448

And to complicate things further your device seems to use a different endianess.

bonnybabukachappilly commented 2 years ago

Can you share Omron's document if any?

bonnybabukachappilly commented 2 years ago

when switching to programming mode, and paring I am getting a response of b'\x82\x0f' + b'\x00'18 instead of b'\x82' + b'\x00'16

userx14 commented 2 years ago

when switching to programming mode, and paring I am getting a response of b'\x82\x0f' + b'\x00'18 instead of b'\x82' + b'\x00'16

Oh, so maybe the number of zeros can vary then. Please try the new version and see if it works now.

LazyT commented 2 years ago

0x820F happens for me, if the device isn't paired in OS.

bonnybabukachappilly commented 2 years ago

When connecting to the device after paring, windows will give a prompt to pair. if paired or not will get 'Same transmission failed 5 times, abort' when trying 'again bleak.exc.BleakError: Could not get GATT services: Unreachable' so I need to remove the device from windows, and the same when repeating

bonnybabukachappilly commented 2 years ago

if i give await bleClient.write_gatt_char(self.deviceUnlock_UUID, b'\x01' + b'\x00'16, response=True) insted of await bleClient.write_gatt_char(self.deviceUnlock_UUID, b'\x02' + b'\x00'16, response=True)

I get a 0x8104

userx14 commented 2 years ago

When connecting to the device after paring, windows will give a prompt to pair. if paired or not will get 'Same transmission failed 5 times, abort'

Does the initial pairing with -p work for you? Does the programm print something like "Paired device successfully with new key..."?

when trying 'again bleak.exc.BleakError: Could not get GATT services: Unreachable' so I need to remove the device from windows, and the same when repeating

I also have this issue, sometimes removing the device from the windows bluetooth settings seems to be neccesary

bonnybabukachappilly commented 2 years ago

I can't say paring work, there is 0x01 and 0x02 available with program mode, if 0x02 get a response of 0x820F and for 0x01 0x8104

userx14 commented 2 years ago

if i give await bleClient.write_gatt_char(self.deviceUnlock_UUID, b'\x01' + b'\x00'16, response=True) insted of await bleClient.write_gatt_char(self.deviceUnlock_UUID, b'\x02' + b'\x00'16, response=True)

I get a 0x8104

I think when you write to the unlock UUID the first byte indicates what to do. At least for my device it works like this:

If some bit in the second byte of the response is set, it seems to indicate some error. What kind of error you get in the response when you send something to the unluckUUID also seems to depend on previous commands / errors, so make sure to powercycle your device in between tests.

userx14 commented 2 years ago

Also @LazyT has the same device and the initial pairing also didn't work. If it doesn't work after power cycling maybe you could capture the bt communication with the omron app for the initial pairing and a subsequent data readout after power cycling the device. Android is capable of logging all bluetooth transmission if you enable bluetooth hci snoop log in the developer settings.

LazyT commented 2 years ago

Tested with internal (intel) and external (csr) bluetooth adapter with original mac: doesn't work Tested with the same external bluetooth adapter and cloned android mac: works On android every device works.

That was the reason, why ubpm only works with the faked mac...

userx14 commented 2 years ago

On android every device works. That was the reason, why ubpm only works with the faked mac...

Does that mean you have tested with a Android Phone, that has bluetooth mac address that is different from the one that was originally paired with the omron device?

In starting from Wednesday this week I should have access to a hem-7361t / m500 and can do some testing regarding the pairing process.

LazyT commented 2 years ago

Yes, paired on tablet and works on handy too.

Tried also with cloned mac last byte +1: doesn't work on pc.

So it's not vendor specific blacklist or something like this.

userx14 commented 2 years ago

Ok, I have finished the record decoding function for hem-7361t. Capture

Regarding the pairing, I still find it strange that sending 0x01 (16 0x00) works for you for unlocking the device, even with the correct mac address. When I do the same on my device I get 0x8101 (15 0x00) indicating an error. It only works with the correct unlock code, so there seems to be some hardware level difference between the devices.

bonnybabukachappilly commented 2 years ago

@userx14 Tried to run your code but not pairing. Return code is '820f'

userx14 commented 2 years ago

Yes, paired on tablet and works on handy too. ...doesn't work on pc.

I think @LazyT is right, and it seems to be OS dependent.

I have connected a factory new HEM-7361T to an android device with the following steps: Paired it with the OS Bluetooth dialog, and used nRF-Connect to send the commands. 0x02, then 0x00(16* unlock key) and it responded with the expected codes. If I remove the pairing, and retry the same steps again it responds to the key write request (0x02) with 0x820f.

I always expected that there is a normal bluetooth connection mode, like there is with the hem-7322t, where you short press the blutooth button, and it goes into a mode where the little square directely appears and not the P. But the 7361 does not seem to have that. Instead I have to go into the P mode again, but it works the same from thereon. Send 0x01(16*bytes unlock key) and it changes to the square mode and accepts the commands and I'm able to access the eeprom (manually by with nRF Connect).

I'm currently investigating why it is not working on windows. My programm does not seem to create a pair dialog with this devcies, even if I insert .pair(). Therefore I always get 0x820f, because the device is at this point in time not correctely paired. When I pair the device with the os first, there seems to be a different issue, where the tool does not have access to the gatt services anymore.

userx14 commented 2 years ago

It has something to do the encryption or the pairing procedure for ble devices. Encrypted services seem to be not supported at all by bleak, therefore it is most likeley unusable for this device. The response will therefore be always zero bytes when reading a characteristic and writes will not work at all.

Since the device works with various Android phones and nRF-Connect I also tested with nRF-Connect and their nRF52840 chip using their Windows Desktop Application. There the device also returns 0x820f after connection. But when pairing the device without bonding the device suddenly works and I get the expected responses. So connecting then pairing is needed, which encrypts the btle connection and exchanges keys.

After some experimenting I was able to reproduce the connection steps with bluetoothctl on linux by manually typing the commands and it works too. So here it my log of what I typed in the shell. Not every command might be neccesary, e.g. I suspect that the btmgmt commands are optional, but please try what is required on your devices to get it to work and report your findings.

@LazyT would it be possible to "port" these commands to UBPM, since it uses the bluez stack directely?

I'm currently looking for alternative python libraries which support encrypted bluetooth le services to get this code working with the hem-7362t, but that could mean that the support for windows 10 will no longer be present.

#Test system
#Debian 11 - Kernel 5.10.0-15-amd64
#bluez version: 5.55-3.1
#bluez-firmware version: 1.2-4
#chipset: marvell 88W8897

#mac of omron 28:FF:B2:EE:EE:EE

#remove any previous bonds/pairings to device from the bluetooth settings
sudo su -
btmgmt connectable on
btmgmt bondable on
btmgmt io-cap 3
bluetoothctl --agent NoInputNoOutput
#start the device in p mode
scan on
#wait 2 seconds until you see at least one message from the device
scan off
connect 28:FF:B2:EE:EE:EE
pair 28:FF:B2:EE:EE:EE
menu gat
select-attribute b305b680-aee7-11e1-a730-0002a5d5c51b
notify on
write 0x02
write 0x001234123412341234123412341234123412341234
bonnybabukachappilly commented 2 years ago

Which python library, I have tried Pyatt and it's good. But I am not getting exceptions and no response. while I get it on bleak. Hope it will work for you. Please keep updated.

LazyT commented 2 years ago

would it be possible to "port" these commands to UBPM, since it uses the bluez stack directely?

For over a year I made a bluetooth test tool and all the code was already there, with pairing.

I don't know why it wasn't working, but it works now. I don't understand it.

Didn't test reading records yet, but I guess this will work too.

Holy shit, lol...

userx14 commented 2 years ago

would it be possible to "port" these commands to UBPM, since it uses the bluez stack directely?

For over a year I made a bluetooth test tool and all the code was already there, with pairing.

I don't know why it wasn't working, but it works now. I don't understand it.

Didn't test reading records yet, but I guess this will work too.

Holy shit, lol...

I would guess that they changed something with later bluez versions, btle encryption support seems to be a recent development. Or maybe you executed some of the btmgmt commands and they changed some state of the bt adapter?


Which python library, I have tried Pyatt and it's good.

I have looked into pygatt, but unfortunatelly it seems to be a wrapper around gatttool which is deprecated. Also gatttool when used manually did not work for me when testing.

For python it seems to work with python-bluezero with some trickery for pairing. https://github.com/ukBaz/python-bluezero I will push an update to the python script soon, but that means "bye bye windows support" at least for pairing with the device, unless someone finds a bluetooth librarary for BTLE which support pairing with security 1 type 2.

userx14 commented 2 years ago

Hi, please check out the latest branch which makes it possible to pair and read data from the hem7361t.

https://github.com/userx14/omblepy/tree/bluezeroVersion

When testing with the device I discovered it has some quirks in the protocol which differ from the hem7322t:

Additional observations:

bonnybabukachappilly commented 2 years ago

I tried the new code. When I try that I am getting dbus.exceptions.DBusException: org.bluez.Error.AuthenticationFailed: Authentication Failed

userx14 commented 2 years ago

I tried the new code. When I try that I am getting dbus.exceptions.DBusException: org.bluez.Error.AuthenticationFailed: Authentication Failed

With which linux distro are you running your tests? I am testing with this Debian-11.4 KDE live iso: https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/debian-live-11.4.0-amd64-kde.iso

sudo apt update
sudo apt install python3-pip
pip3 install terminaltables bluezero
git clone https://github.com/userx14/omblepy/
cd omblepy
git checkout bluezeroVersion
python3.9 ./omblepy.py -p -d hem-7361t-d

Testing on Ubuntu 22.04 did not work at all and gave the org.bluez.Error.AuthenticationFailed error.

Ubuntu 20.04 worked but with the following limitations:

Best, Benjamin

userx14 commented 2 years ago

I think the the authentification bug is now also fixed in the windows = crossplatform = bleak version. It was a combination of pairing and reading a characteristic instead of using callbacks and assuming that delay would work there. Pairing is expected to work, but the readout for hem-7361 will not work yet, because the difference for empty readouts (8byte empty response instead if 0xff).

Toei79 commented 1 year ago

it can work with only bluetooh devices without usb ports? BP7450 , looks like it need be a sophisticated model.

i have those models HEM-7342T-Z And HEM 71551T-Z , BP5250

Toei79 commented 1 year ago

i get this https://www.ebay.com/itm/113834640107

looks like one you use but , i tried on win 10 with bluettoh changer without success

userx14 commented 1 year ago

it can work with only bluetooh devices without usb ports? (from your other post:) I have impression it only works on models with both usb port and bluetooth

This tool does only use bluetooth for communication, there is no usb support. So, there is no requrirement for a devices to have an USB port to work with omblepy.

i have those models HEM-7342T-Z and HEM 71551T-Z

HEM-7342T-Z from the outside looks simmilar to the supported HEM-7361t. And your hem-7151t looks a bit like the supported hem-7155t. Maybe it will work with HEM-7361t or HEM-7155t respectivly as the target device. Also I suggets using the "--loggerDebug" flag and see how far in the communication process you get. Since I do not have these devices for testing you would have to do some tests to find out on which adresses the records are stored and create a device specific file like these ones. But I think we should open a seperate github issue for discussing device support to keep things organized.

looks like one you use but , i tried on win 10 with bluettoh changer without success

Changing the mac address of the bluetooth adapter is only required for ubpm, while omblepy shoud include the neccesary pairing routine to work without altering the mac address. The stick you bought uses the special csr harmony driver on windows and from my testing I was NOT able to get it to work with any omron device under Windows. The stick is recommended by UBMP because it is one of the few bluetooth sticks where you can modify the mac-address under linux, but omblepy does not need this feature and this stick does not work on windows. On Linux, you have to use a specific version of the bluetooth stack and other requirements, so it's quiet cumbersome. Therefore if you are using windows I recommend using a different bluetooth chipset which does not require sepecial bluetooth stacks, maybe you already have a builtin bluetooth module in some laptop available.

Toei79 commented 1 year ago

it can work with only bluetooh devices without usb ports? (from your other post:) I have impression it only works on models with both usb port and bluetooth

This tool does only use bluetooth for communication, there is no usb support. So, there is no requrirement for a devices to have an USB port to work with omblepy.

i have those models HEM-7342T-Z and HEM 71551T-Z

HEM-7342T-Z from the outside looks simmilar to the supported HEM-7361t. And your hem-7151t looks a bit like the supported hem-7155t. Maybe it will work with HEM-7361t or HEM-7155t respectivly as the target device. Also I suggets using the "--loggerDebug" flag and see how far in the communication process you get. Since I do not have these devices for testing you would have to do some tests to find out on which adresses the records are stored and create a device specific file like these ones. But I think we should open a seperate github issue for discussing device support to keep things organized.

looks like one you use but , i tried on win 10 with bluettoh changer without success

Changing the mac address of the bluetooth adapter is only required for ubpm, while omblepy shoud include the neccesary pairing routine to work without altering the mac address. The stick you bought uses the special csr harmony driver on windows and from my testing I was NOT able to get it to work with any omron device under Windows. The stick is recommended by UBMP because it is one of the few bluetooth sticks where you can modify the mac-address under linux, but omblepy does not need this feature and this stick does not work on windows. On Linux, you have to use a specific version of the bluetooth stack and other requirements, so it's quiet cumbersome. Therefore if you are using windows I recommend using a different bluetooth chipset which does not require sepecial bluetooth stacks, maybe you already have a builtin bluetooth module in some laptop available.

hey thanks for deep answer , i tried script but no dice, but i gonna check with debug part on linux to see what shows.

i have two usb bluetoth , so this one doesnt works , but it was that path what usb bluetoth you recommend? i cant know about chipset at least online, for what i understand, im in usa.

thats means there its more chances with old laptop to get it work? i mean with the clone mac addreses, kinda like trial and error thing?

userx14 commented 1 year ago

i tried script but no dice, but i gonna check with debug part on linux to see what shows.

This debugging feature is also available on Windows. I would expect the script to fail at some stage, because I suspect that the blood pressure records are stored at different addresses.

what usb bluetoth you recommend?

If you are on windows please try to use an bluetooth dongle / builtin card without some special bluetooth stack. Intel / Marvell / Realtek chipsets worked for me.

I only came across one single dongle that does not work on Windows, it was based on a csr8510 and required installing the CSR Harmony Stack. I think the ebay link you sent is exactely this csr stick which will not work with omblepy on Windows.

i mean with the clone mac addreses, kinda like trial and error thing?

The mac address cloning / changing step is NOT needed for omblepy.

Toei79 commented 1 year ago

ok i tried this bluetooth device and other one one in the pic. i did paring mode on omron device.

image

image

gotcha what you say them they dont documented too much about just mention the three bluetooth devices , i never know about linux only.

i have a laptop from 2012 im going to give a try there.

userx14 commented 1 year ago

Hi, the error in your first screenshot is not related to bluetooth communication.

It has occured because the tool uses the device name after the -d flag to load the device specific programm file with this name from here. So if run it with "-d hem-7342" it searches for hem-7342.py there and aborts. I have so far only determined the addresses for four omron devices, but some of them are quiet simmilar.

Therefore my recommendation from the previous post was to just pass one of the supported devices which is most simmilar to your device after the -d flag: (-d HEM-7361t).

Regarding the second screenshot, you do not need to change the mac address for omblepy.

Toei79 commented 1 year ago

I get this

image

edit

Toei79 commented 1 year ago

ok i press bluetooh button on both omron then i get this

image

Toei79 commented 1 year ago

i move to windows and i connected

image

nice now i need try with ubpm , thats easy? im reading here and see

Toei79 commented 1 year ago

i able to connect but after i get this image without the -p , at first time i get csv after this

try again and dont show up here then a display dialog appear from bluetooth on windows i click accept and on ubpm i started to import data, csv dont compatible with ubpm. sorry many dumb question its like learning.

userx14 commented 1 year ago

Hi,

nice, looks like the initial pairing worked.

But when you did the second test without the -p flag it has lost connection to the device.

try again and dont show up here then a display dialog appear from bluetooth on windows i click accept

Have you enabled the normal bluetooth communication by shortly pressing the bluetooth button on your omron for this test? A small o should be displayed on the omron's screen. Also the timeout of the omron is not that long, meaning you have to relatively quickly connect after enabling the bluetooth mode. You can also try some of the troubleshooting steps from the readme, like unpairing the device in the windows bluetooth settings, the bluetooth pairing is independent from the omron pairing procedure (-p), so you do not have to use the -p option from now on.

nice now i need try with ubpm , thats easy?

Well I think the device is also not supported by ubpm. Ubpm and omblepy are simmilar tools, with ubpm having a nice gui and database to store the records, while omblepy aims to be a simple command line tool to just dump the records straight to a csv file.

ubpm i started to import data, csv dont compatible with ubpm.

Most likeley, no records have been sent yet, so the csv file is expected to be empty. Probably that's why ubpm rejects reading it.

To see some more info could you add the --loggerDebug option when running the tool?

Toei79 commented 1 year ago

ok when the first tries that no happened and runs normal even without -p , when i did i able to get .csv files one with some data that i can see on excel but not on ubpm .

maybe get those erratics because i was searching on bluetooth windows? maybe? that was after , just see bp250 i guess on windows bluetooth so i tought was fine,

i keep trying and run script then get those erratics on command line, but at same time again a popup appears the windows bluetooth showing connect to bp250 , what i did, at same time i have ubpm open then it find the blood pressure and donwload all data, and display all stuff perfectly

Toei79 commented 1 year ago

when its pairing by windows its not keep for longer

Toei79 commented 1 year ago

im going check with debug tonight, its that normal or its not fine? yea i press bluetooth button on omron device

if keep getting trouble i do those steps on readme

user1.csv

this its file that i get from omron , but dont load fine on ubpm.

userx14 commented 1 year ago

im going check with debug tonight, its that normal or its not fine? yea i press bluetooth button on omron device

if keep getting trouble i do those steps on readme

user1.csv

this its file that i get from omron , but dont load fine on ubpm.

Oh, I just remembered that ubpm does not need the .csv file, it needs ubpm.json which should also be generated by omblepy automatically. The csv is for using something like Excel or the plotCsv.py visualizer from omblepy.

when its pairing by windows its not keep for longer

Sorry, I'm not really sure what you mean by that, can you elaborate?

The bluetooth connection process is a bit finicky, for me it works best if the device is not paired / stored in the windows bluetooth settings to start. You can use the "-m" option with your omron's bt-mac address to skip the scanning for the device, which makes it easier to stay within the timeout of the omron for connecting. Only accept the bluetooth pairing popup dialog from windows when it shows up directely after starting the tool.

Toei79 commented 1 year ago

image image

with debug . i tried with HEM 7342

Toei79 commented 1 year ago

Model HEM7150T looks working image

Toei79 commented 1 year ago

i tried connect on linux but i get the same error, i dont connect nothing yet on linux, so i tried there image

image i tried all those on command line

python3 ./omblepy.py -p -d HEM-7155T --loggerDebug python3 ./omblepy.py -p -d HEM-7322T --loggerDebug python3 ./omblepy.py -p -d HEM-7361T --loggerDebug python3 ./omblepy.py -p -d HEM-7600T --loggerDebug image image

Toei79 commented 1 year ago

sorry for the mess here lmk if i deleted all this messages, i just reading all messages, and then try open one.

Toei79 commented 1 year ago

i have other omron device, one with ekg reading , model its BP7900 , HEM-7530T-Z

i rereading here, its a way to get needed info to runs those with omblepy? so far i have two devices

i can try here just idk where start