search

cyberman54 / ESP32-Paxcounter

Wifi & BLE driven passenger flow metering with cheap ESP32 boards
https://cyberman54.github.io/ESP32-Paxcounter/
Other
1.75k stars 408 forks source link

Integrate German "Corana Warn App" BLE Counter #615

Closed erniberni closed 4 years ago

erniberni commented 4 years ago

The German Corona App reports with a well defined BLE Service UUID, that could be detected and counted. It would be a nice feature to count wifis and this BLE Service UUID to determine the ratio of installed corona apps per smartphones. The solution is already used in this repo: https://github.com/kmetz/BLEExposureNotificationBeeper The central function is advertisedDevice.getServiceUUID().equals(BLEUUID((uint16_t) 0xfd6f)

FlorianLudwig commented 4 years ago

I have been looking into the same thing. Relevant docs: https://blog.google/documents/58/Contact_Tracing_-_Bluetooth_Specification_v1.1_RYGZbKW.pdf

Take away:

AugustQu commented 4 years ago

I started playing with the code mentioned above (code by kmetz).

Here is what I got in the serial-monitor: 13:10:17.284 -> Hi. 13:10:17.913 -> Scanning ... 13:10:17.913 -> ----- 13:10:22.916 -> Count: 0 13:10:22.916 -> ----- 13:10:24.613 -> + 59:8c:f4:c1:85:91 13:10:24.613 -> BEEP 13:10:27.944 -> Count: 1 13:10:27.944 -> ----- 13:10:29.742 -> ... 59:8c:f4:c1:85:91 13:10:32.933 -> Count: 1 ..... 13:10:42.944 -> ----- 13:10:47.957 -> Count: 1 ..... 13:12:37.966 -> ----- 13:12:42.987 -> - 59:8c:f4:c1:85:91 13:12:42.987 -> Count: 0 13:13:23.012 -> ----- 13:13:24.638 -> + 59:8c:f4:c1:85:91 13:13:24.638 -> BEEP 13:13:27.994 -> Count: 1 13:13:27.994 -> ----- 13:13:29.820 -> ... 59:8c:f4:c1:85:91 13:13:33.016 -> Count: 1 .....

My first question is: is the value "59:8c:f4:c1:85:91" the token the CWA produces?

More question will come when I port this code to the paxcounter-source.

AugustQu commented 4 years ago

ah, so it looks better:

13:10:17.284 -> Hi. 13:10:17.913 -> Scanning ... 13:10:17.913 -> ----- 13:10:22.916 -> Count: 0 13:10:22.916 -> ----- 13:10:24.613 -> + 59:8c:f4:c1:85:91 13:10:24.613 -> BEEP 13:10:27.944 -> Count: 1 13:10:27.944 -> ----- 13:10:29.742 -> ... 59:8c:f4:c1:85:91 13:10:32.933 -> Count: 1 ..... 13:10:42.944 -> ----- 13:10:47.957 -> Count: 1 ..... 13:12:37.966 -> ----- 13:12:42.987 -> - 59:8c:f4:c1:85:91 13:12:42.987 -> Count: 0 13:13:23.012 -> ----- 13:13:24.638 -> + 59:8c:f4:c1:85:91 13:13:24.638 -> BEEP 13:13:27.994 -> Count: 1 13:13:27.994 -> ----- 13:13:29.820 -> ... 59:8c:f4:c1:85:91 13:13:33.016 -> Count: 1 .....

AugustQu commented 4 years ago

ok, I learned something: it's the advertiser address, which is 48 bits long. It is valid between 10 and 20 minutes and changes then.

AugustQu commented 4 years ago

yesterday I tested the code mentioned above and it works. It counted the correct number (I was on a public plac and I asked the people there about the Corona-app installed on their smartphones).

So: does it make sense to implement the code (the feature) into the Paxcounter-code?

a hint: look for the app Ramble in the Google play store. It does already this (and more). Shall we include this functionality into the Paxcounter-code? Or shall we say: use 2 programs?

Found this app via this text: https://www.faz.net/aktuell/politik/inland/datenrecherche-zu-corona-app-fehler-im-system-16892844.html

erniberni commented 4 years ago

I would really appreciate it if this option would be build into the code. You can add an optional switch which - if BLE is enabled - counts either the BLE addresses or the Corona App IDs. If this is not included in the code - could you make the code public?

AugustQu commented 4 years ago

Hi erniberni,

the code I used for my test is already published: https://github.com/kmetz/BLEExposureNotificationBeeper My changes included the removal of the LED- and beeper-related stuff.

I will look into porting the code to Paxcounter.

AugustQu commented 4 years ago

a preliminary remark: I never worked in the Bluetooth-environment, only used it sometime. So I'm just entering this topic. Maybe I'm wrong for my proposals.

@cyberman54 In the file blescan.cpp the function gap_callback_handler() is called for every BLE-device that is detected. The structure scan_rst contains all the information needed for detecting an active CWA (Corona Warn App).

My approach: Work as it is coded now. Plus add some lines for checking for the signature 0xFD6F (=CWA) The data needed can be found in:

Question: is this approach OK?

Another Question: in the code mentioned in the beginning of this thread the code given as an example looks like: if (!advertisedDevice.haveServiceUUID() || !advertisedDevice.getServiceUUID().equals(BLEUUID((uint16_t) 0xfd6f)) ) { // this is not a CWA }

It uses classes like BLEDevice, BLEScan, BLEAdvertisedDeviceCallbacks, .... This way looks much simpler (at least to me) then the code used in tha callback-funciton in the Paxcounter-src. Why does the code in Paxcounter does not use this approach? Is there anything the code here checks that is not possible with these classes? Or was the code here written before these classes were available?

cyberman54 commented 4 years ago

Reason was to keep the code sticking to native ESP32 API, to not use third party BLE libs, because i found some of them very resource hungry. Result was, that paxcounter binary exceeded the limit of flash memory and thus was no more flashable.

It shouldn't be very difficult to filter the CWA UUID only by using the ESP32 BLE API.

Using callbacks for examining BLE packets is essential to avoid blocking code triggering the watchdog.

AugustQu commented 4 years ago

I think I can do this.

Then we have to numbers for BLE-devices. 1 for the devices found, 1 for the devices with the CWA installed. How do we present the numbers:

I think in all 3 cases this should (and can be) done. In which form?

cyberman54 commented 4 years ago

@AugustQu Data transfer: My suggestion is to handle the CWA counter like a user sensor. Thus, using one of the three user sensor values in the payload protocol to transfer it via LoRaWAN. Display: a separate simple display page should be added (easy, since the display code is already structured in pages). SD storage: add to your format.

AugustQu commented 4 years ago

@cyberman54

regarding user sensor: when I enable it it sends data to TTN in this form: port 10 payload 01 02 03 port 11 payload 01 02 03 port 12 payload 01 02 03

OK, can change this

What I need is a define that enables only one sensor. Currently we have HAS_SENSORS which enables all 3. I will change this to HAS_SENSOR_1, HAS_SENSOR_2 and HAS_SENSOR_3.

Agreed?

Storing the dataon the SD-card works, sending it to TTN will work soon. Display is not started now.

cyberman54 commented 4 years ago

Agreed!

AugustQu commented 4 years ago

@cyberman54 I started with the code. I will do some more tests but I will upload the modifications to my branch of the Paxcounter.

Can you please create a branch CWA here so I will make a pull-request against this branch. Thanks.

PS: what is missing now:

AugustQu commented 4 years ago

@cyberman54

I encountered a problem with the latest version of the code. The SD-card-handling is broken (I assume).

I am using the ttgov21new-board and the values for SDCARD_CS etc. are not set correctly.

After modifying the file sdcard.h (quick and dirty) it did work again.

I think these belong into the file hal/ttgov21new.h (again).

Also the value of HAS_SDCARD is set to 2 in this file which is not the correct value, at leastnot for the board I use. It has to be 1.

AugustQu commented 4 years ago

@cyberman54

another problem on the TTN-side: I compiled the code with:

WIFICOUNTER 0 BLECOUNTER 1

so only Bluetooth-devices will be counted. On the TTN-side the value 06 00 appears. The JS-code intereprets this as wifi: 6 (but it's the value of blth-devices).

cyberman54 commented 4 years ago

SDCARD: I added two different SD interfacing modes. This is necessary because different boards use different modes. Probably wrong settings for TTGOV21, please feel free to correct.

Payload decoder: Yes, this a bug in the decoders.

AugustQu commented 4 years ago

@cyberman54 I created a pull-request.

For testing it you have to modify the paxcounter.conf file:

I used the ttgov21new.h-file for my device.

Please give it a try.

AugustQu commented 4 years ago

Open topics:

sbamueller commented 4 years ago

can't we filter for a specific beacon?

On Tue, Sep 1, 2020 at 4:36 PM August Quint notifications@github.com wrote:

Open topics:

-

the README as usual

the display-code is a first shot. It looks ugly if you count wifi- and bluetooth-devices together.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cyberman54/ESP32-Paxcounter/issues/615#issuecomment-684900915, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFZK2AD7DJXEE24GVK7QFW3SDUBGJANCNFSM4OLMSIQA .

-- Sebastian Müller

www.sbamueller.de sbamueller@googlemail.com skype: sbamueller

tel: +49 (0)761 27 77 60 fax: +49 (0)761 89 72 1 39 mobil: +49 (0) 175 1 55 44 95

AugustQu commented 4 years ago

@sbamueller

Sorry, don't understand your argument.

In my tests I counted the number of bluetooth-devices and reported this number as BLTH. Of these devices I checked for the signature 0xFD6F which is reserved for the Corona Warn App (CWA). This number is reported with the label CWA on the display and on the SD-card.

In blescan.cpp you find the pattern:

// checking for CWAs we need this magic bytes: static const char cwaMagicBytes[] = "\x03\x03\x6F\xfd";

If the beacon you want to count sends another pattern simply change this. If it sends a pattern in another packet you have to change a bit more.

AugustQu commented 4 years ago

I used this document: https://blog.google/documents/58/Contact_Tracing_-_Bluetooth_Specification_v1.1_RYGZbKW.pdf

On page 3 under the header Contact Detection Service you find a description of the packet my code is looking for.

I'm new to the concept of these beacons, never used one, never read about the inner workings of these devices. So I can only guess: if they announce their presence in the same way one only has to change what I called the MagicBytes and it should work.

Sorry, can't test it here because I don't have acces to such a thing.

cyberman54 commented 4 years ago

Thanks for this contribution. I pushed this to separate branch CWA and made some minor modifications. Before merging with master we should test this a while. Until now i was not able to catch and count a CWA UUID beacon. So im am not sure yet if the approach is working.

AugustQu commented 4 years ago

@cyberman54

testing: agreed.

Even if you do not catch any App you can look how I positioned the information on the screen. When Wifi and Bluetooth are enabled this looks very ugly.

Here is a result of my latest test in Wiesbaden on the market:

date, time, wifi, bluet,cwa 00.00.1970,00:01:10,0,172,7 00.00.1970,00:02:10,0,141,6 00.00.1970,00:03:10,0,158,10 00.00.1970,00:04:10,0,126,4 00.00.1970,00:05:10,0,139,5

VENDORFILTER is disabled. There are a lot of Bluetooth-devices. Don't know if the numbers are correct.

PS: I did this test with the code I've uploaded.

cyberman54 commented 4 years ago

For some unknown reason i was not yet able to catch a CWA beacon with current code in CWA branch. Not sure if it's working. Is anyone in a dense environment and can test?

AugustQu commented 4 years ago

@cyberman54

Hmmmm, later I will download the CWA-branch from here and try it.

With my version (taken from the CWA-branch of my fork):

I will do some more tests today.

PS: did you set VENDORFILTER to 0? Please do so. Otherwise I did not see the Smartphones.

AugustQu commented 4 years ago

@cyberman54 you may also use the code @erniberni mentioned in the start of this thread. Just for comparing the results.

AugustQu commented 4 years ago

I took a copy from the CWA-branch here and made these modifications to it:

compiled it and let it run. Here is what I've seen in the serial monitor:

[D][macsniff.cpp:132] mac_add(): known BLTH RSSI -88dBi -> salted MAC C55392FB -> Hash C3D4 -> WiFi:0 BLTH:3 (CWA:0)-> 122884 Bytes left [D][macsniff.cpp:132] mac_add(): new BLTH RSSI -87dBi -> salted MAC 6E6EB2B9 -> Hash 87AB -> WiFi:0 BLTH:4 (CWA:0)-> 122848 Bytes left [D][macsniff.cpp:132] mac_add(): new BLTH RSSI -88dBi -> salted MAC AD937F19 -> Hash 0F2A -> WiFi:0 BLTH:5 (CWA:1)-> 122776 Bytes left [D][macsniff.cpp:132] mac_add(): known BLTH RSSI -91dBi -> salted MAC C55392FB -> Hash C3D4 -> WiFi:0 BLTH:5 (CWA:1)-> 122776 Bytes left

So you can see it found an app (the one on my smartphone).

I also had 2 devices in my list. This can happen if the mac-address is changed so a knew (=unknown) mac-address appears and will be hashed and stored.

PS: When you change the label from CWA to ENS you have to change the output accordingly. With the next version I will do this.

cyberman54 commented 4 years ago

Yes, now i can confirm: it works.

cyberman54 commented 4 years ago

Done. Thanks to all contributors!

AugustQu commented 4 years ago

Please let me continue the discussion here.

Yesterday I made another test on a public place. I asked the people there: who has the Corona App on the smartphone? And I got 8 persons with this app. So I started my test:

The results of the first and second programs are OK, but the number counted with the Paxcounter code is not correct. I try to figure it out.

Can someone please help in testing the correctness of the code.