Internet of Things data protocol - simplistic text and CSV-friendly IoT data protocol. Options: Schema-less, schema+versioning, building deltas for data values. Easy to re-use.
We can see a lot off complex protocols in IoT and IIoT world. Sometimes it's like overengineering for IoT hobby solutions. This is text protocol with simple syntax, readable and self-explanaible rules.
We have a few base facts about this new IoText
data protocol:
IoText data protocol
, not network or binary encrypted protocol, so we not expected super performance at start point.IoText data protocol
for every your solutions, when you need sending data from devices, e.g. Edge Devices to the cloud.Text data protocol
we can encapsulate on top of other more perfect IoT protocols like MQTT, AMQP, and NATS.We have simple concept like CSV, comma separated data items. One row have all metrics with one:
,
- comma separation of data items and metrics:
- data types and values separator|
- command types and command context separator=
- split data item charWe have a few types of items possible inside message:
You can see reference implementation of item types in ItemTypes
class in src/types
package.
t|3900237526042,d|device_name_001,m|val_water_level1=i:42
We can see informations like:
As you imagine, it's simple as possible ;-)
We have a few basic item types:
t
- timestamp UNIX milis itemd
- device name itemm
- metric itemh
- health checkc
- CRC-16 (from MODBUS control sum) for all combined IoText payload as stringYou can see reference implementation in ItemTypes
class in src/types
package.
We have a few basic data types:
i
- integerd
- decimal (not float!)b
- booleant
- textI
- list of integersD
- list of decimals (not float!)B
- list of booleansT
- list of texts (base64 encoded with UTF-8, removed and auto-restored padding at the end - conflict with "=" protocol separator)You can see reference implementation in MetricDataTypes
class in src/types
package.
We have a few basic data codecs:
MetricDataItemCodec
- decode/encode methods for full coding proces for data rowItemCodec
- decode/encode methods for data row itemIoTextCodec
- decode/encode methods for data row items with Python data structures full support (you can use linters and debugging with no risks)You can see reference implementation in src/codecs
package.
You need two required informations:
h
item type for helth check, too).IoText protocol in schema-less version
data row example:
t|3900237526042,d|device_name_001,m|val_water_001=i:1234,m|val_water_002=i:15,m|bulb_state=b:1,m|connector_state=b:0,m|temp_01=d:34.4,m|temp_02=d:36.4,m|temp_03=d:10.4,m|pwr=d:12.231,m|current=d:1.429,m|current_battery=d:1.548
and after de-serialization to Python data structures we can see:
[
Item(kind='t', name='3900237526042', metric=None),
Item(kind='d', name='device_name_001', metric=None),
Item(kind='m', name='val_water_001', metric=MetricDataItem(data_type='i', value=1234)),
Item(kind='m', name='val_water_002', metric=MetricDataItem(data_type='i', value=15)),
Item(kind='m', name='bulb_state', metric=MetricDataItem(data_type='b', value=True)),
Item(kind='m', name='connector_state', metric=MetricDataItem(data_type='b', value=False)),
Item(kind='m', name='temp_01', metric=MetricDataItem(data_type='d', value=Decimal('34.4'))),
Item(kind='m', name='temp_02', metric=MetricDataItem(data_type='d', value=Decimal('36.4'))),
Item(kind='m', name='temp_03', metric=MetricDataItem(data_type='d', value=Decimal('10.4'))),
Item(kind='m', name='pwr', metric=MetricDataItem(data_type='d', value=Decimal('12.231'))),
Item(kind='m', name='current', metric=MetricDataItem(data_type='d', value=Decimal('1.429'))),
Item(kind='m', name='current_battery', metric=MetricDataItem(data_type='d', value=Decimal('1.548')))]
IoText protocol in schema-less version
data row example with lists values:
t|3900237526142,d|device_name_002,m|ints_list=I:+1-22+333333,m|bools_list=B:0111,m|decimals_list=D:-123.456+1234567890.98765+999.8,m|texts_list=T:Wyd8JywgJzphYmMnLCAnISFAJywgJ3h5MHonLCAnMWFiYywnXQ
and after de-serialization to Python data structures with lists values we can see:
[
Item(kind=ItemTypes.TIMESTAMP_MILIS, name="3900237526142", metric=None),
Item(kind=ItemTypes.DEVICE_ID, name="device_name_002", metric=None),
Item(
kind=ItemTypes.METRIC_ITEM,
name="ints_list",
metric=MetricDataItem(data_type=MetricDataTypes.INTEGERS_LIST, value=[1, -22, 333333]),
),
Item(
kind=ItemTypes.METRIC_ITEM,
name="bools_list",
metric=MetricDataItem(data_type=MetricDataTypes.BOOLS_LIST, value=[False, True, True, True]),
),
Item(
kind=ItemTypes.METRIC_ITEM,
name="decimals_list",
metric=MetricDataItem(data_type=MetricDataTypes.DECIMALS_LIST, value=[-123.456, 1234567890.98765, 999.8]),
),
Item(
kind=ItemTypes.METRIC_ITEM,
name="texts_list",
metric=MetricDataItem(data_type=MetricDataTypes.TEXTS_LIST, value=['|', ':abc', '!!@', 'xy0z', '1abc,']),
),
]
IoText protocol in schema-less version
data row example:
t|123123123123,d|device_one,m|value=d:123.321,c|4C5A
and after de-serialization to Python data structures we can see:
[
Item(kind=ItemTypes.TIMESTAMP_MILIS, name="123123123123", metric=None),
Item(kind=ItemTypes.DEVICE_ID, name="device_one", metric=None),
Item(
kind=ItemTypes.METRIC_ITEM,
name="value",
metric=MetricDataItem(data_type=MetricDataTypes.DECIMAL, value=Decimal('123.321')),
),
Item(
kind=ItemTypes.CRC,
name="4C5A",
metric=None
)
]
You can use below methods from IoText Python API:
iotext_data_struct = IoTextItemDataBuilder.from_json(iotext_msg_as_json)
builder = IoTextItemDataBuilder(...)
builder.add_measure(...)
builder.to_json()
IoText message as JSON row example:
[{"t": "3900237526042"}, {"d": "DEV_NAME_002"}, {"m": "battery_level", "v": 12.07, "t": "d"}, {"m": "open_door", "v": "1", "t": "b"}, {"m": "open_window", "v": "0", "t": "b"}, {"m": "counter_01", "v": 1234, "t": "i"}, {"m": "ints_list", "v": "-12+13-14", "t": "I"}, {"c": "9838"}]
and after de-serialization from JSON format to IoText's Python data structure:
[
Item(kind=ItemTypes.TIMESTAMP_MILIS, name="3900237526042", metric=None),
Item(kind=ItemTypes.DEVICE_ID, name="DEV_NAME_002", metric=None),
Item(
kind=ItemTypes.METRIC_ITEM,
name="battery_level",
metric=MetricDataItem(
data_type=MetricDataTypes.DECIMAL,
value=Decimal("12.07"),
),
),
Item(
kind=ItemTypes.METRIC_ITEM,
name="open_door",
metric=MetricDataItem(data_type=MetricDataTypes.BOOL, value=True),
),
Item(
kind=ItemTypes.METRIC_ITEM,
name="open_window",
metric=MetricDataItem(data_type=MetricDataTypes.BOOL, value=False),
),
Item(
kind=ItemTypes.METRIC_ITEM,
name="counter_01",
metric=MetricDataItem(data_type=MetricDataTypes.INTEGER, value=1234),
),
Item(
kind=ItemTypes.METRIC_ITEM,
name="ints_list",
metric=MetricDataItem(
data_type=MetricDataTypes.INTEGERS_LIST,
value=[-12, 13, -14],
),
),
Item(
kind=ItemTypes.CRC,
name="9838",
metric=None,
),
]
from src.codecs.iot_ext_codec import IoTextCodec
MSG_1_EXAMPLE = '''t|3900237526042,d|device_name_001,m|val_water_001=i:1234,m|val_water_002=i:15,m|bulb_state=b:1,m|connector_state=b:0,m|temp_01=d:34.4,m|temp_02=d:36.4,m|temp_03=d:10.4,m|pwr=d:12.231,m|current=d:1.429,m|current_battery=d:1.548'''
item = IoTextCodec.decode(MSG_1_EXAMPLE)
print('decoded_msg:', item)
item_encoded_str = IoTextCodec.encode(item)
print('encoded_msg :', item_encoded_str)
assert item_encoded_str == MSG_1_EXAMPLE
from src.builders.iot_ext_item_data_builder import IoTextItemDataBuilder
EXPECTED_MSG = '''t|3900237526042,d|DEV_NAME_002,m|battery_level=d:12.07,m|open_door=b:1,m|open_window=b:0,m|counter_01=i:1234'''
builder = IoTextItemDataBuilder(3900237526042, 'DEV_NAME_002', add_crc16=False)
builder.add_measure('battery_level', 12.07)
builder.add_measure('open_door', True)
builder.add_measure('open_window', False)
builder.add_measure('counter_01', 1234)
built_msg = str(builder)
print("built_msg: ", built_msg)
assert EXPECTED_MSG == built_msg
$ pip install git+https://github.com/bieli/IoText-data-protocol@main#egg=IoText-data-protocol
## test in REPL
$ python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
...
>>> from src.builders.iot_ext_item_data_builder import IoTextItemDataBuilder
>>> b = IoTextItemDataBuilder(123, 'D1')
>>> b
<src.builders.iot_ext_item_data_builder.IoTextItemDataBuilder object at 0x7f9ebbc1d550>
>>> str(b)
't|123,d|D1'
IoText data protocol
on top od MQTT IoT protocol