Parser for arbitrary binary protocols

[eduard93]

Rapid growth of IoT industry in recent years produced a plethora of new protocols with varying levels of standardization and adoption.

Quite a lot of these protocols are binary because:

• It allows for greater throughput
• On a device level these protocols are easier to implement
• On a device level they are more resource (CPU/RAM) efficient

Currently InterSystems IRIS offers several functions for work with binary data:

• zzdump
• $zhex
• $char
• $ascii

But protocol implementation is left as a task for the end user.

Removing the time-consuming need for protocol implementation would allow for easier and stronger positioning of InterSystems IRIS as an IoT platform.

One of the possible approaches to protocol implementation is declarative parsing. With declarative parsing user describes the data structure that he wants parsed (i.e.{ name: title, type: string, length: 5}...) in some declarative language (XML, JSON, YAML) and we use this to generate a concrete parser. Afterwards user passes protocol data to the concrete parser implementation and receives parsed values (as objects or locals or $lb).

Kaitai Struct is an OpenSource (MIT license) declarative parser implementation which allows user to generate parser code for several languages: C++, C#, Go, Java, JavaScript, Lua, Perl, PHP, Python, Ruby, Rust. It allows easy language extensibility by keeping most of the process language-agnostic and only the last step (concrete parser generation) is end-language specific.

I propose we add ObjectScript as a Kaitai Struct generation language. It would allow us to generate native ObjectScript parsers easily which would help with presenting InterSystems IRIS an IoT platform especially at a PoC stage.

[eduard93]

Alternatives to Kaitai Struct:

• binpac - C++ only
• preon - Java only

The advantage of Kaitai Struct is that it's not aimed at one language form the beginning, so adding a new language is an expected procedure.

I will update the list if I find more examples of declarative parsers.

[eduard93]

Article on the topic

[rcemper]

A personal comment: Given the support by Kaitai Struct for such a broad range of widespread mostly modern languages is a strong argument against this. COS / ISOS is neither modern nor widespread and it's as far away from those languages as the Moon from Andromeda Galaxis. In addition, we have excellent interfaces to C++, C#, Java, JavaScript, Python,... As a former CIO, I would rate this a serious failure of my Software Engineering Designer. The pure fact that it is possible, is not enough reason and qualification to do it. Nice to prove engineering and abstraction capability as an academic exercise, but without any commercial value. my 2¢

[evshvarov]

Could you please reintroduce the idea on the InterSystems Ideas portal? Use InterSystems SSO to sign in.