libpax integration

[lifezoned4]

Adds a possible use of the library libpax. The project extracts the device counting code into a reusable library for further use.

Using of this feature can be toggled in the paxcounter.conf with #define LIBPAX 1.

Example switching of BLE and WiFi on runtime is implemented. Changing parameters used in the scanning process on runtime is not. At the moment the library is instantiated with its default configuration values, which should be the default configuration values of the paxcounter project.

The public release v0.1.1 of libpax should be available soon with which this branch will be working.

[cyberman54]

@lifezoned4 @FlorianLudwig @spmrider please note current https://github.com/cyberman54/ESP32-Paxcounter/tree/development with working paxexpress (including some, but not yet all necessary, cleanups) and libpax integration.

[cyberman54]

Sorry, then i was too fast merging it. But it was merged to development branch only, so it should be allright.

[cyberman54]

@lifezoned4 @FlorianLudwig i took a quick look in the libpax code. Not sure, if this is an issue, but i changed this some time ago in the paxcounter code, after observing watchdog resets under heavy duty MAC load (e.g. big crowds of people).

Wifi and BLE sniffing on ESP32 is using callback functions. The packet handler, throwing the MACs to the callback function, is running on PRO CPU (core0), while the callback functions are running in APP CPU (core1) in arduino-espressif32 framework. Thus, race conditions can happen, and i assume that this causes the observed resets.

I solved this in paxcounter code by minimizing the callback function to just adding the received MAC to a queue. This keeps the callback function lightwight and fast, while the CPU time consuming MAC analyzation is carried out to a separate RTOS task on core1 which eats the MACs from the queue and processes it. This ensures asynchronous operation of packet handler and MAC processing in mac_add(). After i made this change, i did not observe further resets under load.

I observed the MAC queue length in operation. Under low load it sticks to zero, meaning that feeding and consuming MACs happen synchronous. Under load the queue grows, i saw maximums of 2 MACs waiting in the queue for processing. That means, without queuing MACs will be lost and there will be a risk of race conditions.

I did not carry out further realtime analysis of this setup, but would suggest to add queuing to libpax. It seems queuing is more important to BLE sniffing but Wifi sniffing, because BLE is much faster than Wifi sniffing.

/cc @spmrider

[FlorianLudwig]

@cyberman54

The data structure how the counting is done differently than in the paxcounter project and therefore it was decided against a queue:

• The libpax has constant RAM requirements (e.g. does not dynamically allocated memory). So no matter how many devices there are the maximum amount of ram set is allocated during compile time and doesn't impact stability during runtime. LIBPAX_MAX_SIZE is the maximum of macs counted. With two bytes per MAC stored that is LIBPAX_MAX_SIZE*2 bytes memory. No additional memory per device is allocated, comparison see memory of STL containers
• CPU usage is a lot lighter than with std:set - the cpu usage per packet does increase with more devices. Calculating the hash of the mac was a major contributor to CPU usage. With only counting randomized mac-adresses and only using the last two bytes, the hash became not needed anymore. Benchmarks:

In our tests, the new structure combined with leaving out the hashing resulted in a higher throughput than including the queue.

Some benchmark code can be found here: https://github.com/dbSuS/libpax/blob/master/test/libpax_benchmark.cpp

With the new structure being "hashmap like" setting LIBPAX_MAX_SIZE higher results in better performance as it decreases the probability of

[cyberman54]

Thanks for the explanations. I see there were deep analyses on this topic, great to know! So we will be able to remove lots of code from paxcounter, once we have libpax integration completed. Well done!

[cyberman54]

@lifezoned4 I continued work on preliminary libpax integration, and finished this work today in development branch. The resulting current version now should be feature complete, and can be merged with master without conflict. But is not fully tested yet (i.e. rcommands for setting sniffing parameters during runtime). I will now suspend this work, waiting for final version of libpax API.

Since libpax scan cycle and senddata cycle now work indepently in parallel, how can we sure that both cycles are synced, meaning libpax finished a counting cycle before senddata tranmits the counter values?

/cc @FlorianLudwig @spmrider

[FlorianLudwig]

we could use the callback on the end of each scan cycle of libpax to trigger the send.

Would that be an option?

[cyberman54]

Yes, i think that's the most straight forward way to sync it.

[cyberman54]

@FlorianLudwig I implemented it. Was amazingly easy.

[FlorianLudwig]

Looks good!

The integration is going quite quickly :) nice

[cyberman54]

Notes missing features after integration of libpax, to be done: [ ] counter mode cumulative [ ] CWA detection [ ] API for current channel (needed for display, currently solved by using a global var)