search

bieli / IoText-data-protocol

Internet of Things data protocol - simplistic text and CSV-friendly IoT data protocol. Options: Schema-less, schema+versioning, building deltas for data values. Easy to re-use.
Apache License 2.0
0 stars 2 forks source link

Optimization for TEXT fields with RLE algo. (https://en.wikipedia.org/wiki/Run-length_encoding) - new sign for TEXT field upper 'T' letter #22

Open bieli opened 7 months ago

bieli commented 7 months ago

RLE is a common algo. and probably nothing special, but in case of data streams with TEXT-type data, we can use RLE to optimize bytes used to transmit data records.

Example text:

WWWWWWWWWWWWBWWWWWWWWWWWWBBBWWWWWWWWWWWWWWWWWWWWWWWWBWWWWWWWWWWWWWW

With an RLE data compression algorithm we can see:

12W1B12W3B24W1B14W

We need to implement a special char in IoText data protocol for this - "T" high letter as RLE compressed data identifier. Example of record in IoText data protocol with suggested field type:

t|3900267526042,d|meter_power_01,m|err_msg=T:12W1B12W3B24W1B14W

in a typical TEXT field type, we will have the below message:

t|3900267526042,d|meter_power_01,m|err_msg=t:WWWWWWWWWWWWBWWWWWWWWWWWWBBBWWWWWWWWWWWWWWWWWWWWWWWWBWWWWWWWWWWWWWW

Little optimizations are not too bad, if we fighting with bytes in the transmission channel - for example in Narrow Band IoT or other thin options/transmission channels. Power consumption, it's a strongly connected aspect for battery-powered IoT devices.

bieli commented 4 months ago

"T" upper letter is reserved for this implemention: https://github.com/bieli/IoText-data-protocol/issues/24 - try special sign from UTF-8 maybe... է