Decoupling visualizer vs parser-running engine (visualizer protocol)

[GreyCat]

Following the discussion at LOR, I'd like to propose the following improvement. Let's make "visualizer" more modular and separate two parts that are actually there in any visualizer (Java, JS, Ruby):

• "visualizer" part — the one that provides GUI, allows user to traverse the parsed object tree, shows hex dump, highlights regions in hex dump
• "engine" part — one that:
  • takes a .ksy as input
  • runs a ksc to compile .ksy -> .java / .js / .ruby (with --debug)
  • optionally, if this requires more compilation (i.e. Java), compiles .java -> .class
  • loads resulting class into some VM / execution space
  • runs parsing on a given file
  • after all that, can answer queries like "give me a list of objects at root node", etc — in our current languages (Java, JS, Ruby) this is solved with a combination of language's reflection API + debugging aids like seq arrays

Pros that we'll get:

• it would be easier to implement a visualizer and/or plug it into any existing hex editor/viewer
• more crash-prone solution with proper isolation: even if some malicious code will get into the "engine" part, it will never do any harm to "visualizer" part
• "visualizer" in any language can run "engine" in any arbitrary language; like Ruby "visualizer" running with a Java engine; this would, for example, allow to load extra language-specific opaque classes and it will still do ok
• "engine" can be potentially re-implemented as interpreter (and no one will notice)
• network transparency, multi-user access, etc

Cons:

• extra programming complexity: need to make a protocol that will be used between these 2 components and follow it
• on a common one-user installation, need to run and coordinate two simultaneous processes → extra hassle

[ams-tschoening]

"visualizer" in any language can run "engine" in any arbitrary language; like Ruby "visualizer" running with a Java engine; this would, for example, allow to load extra language-specific opaque classes and it will still do ok

That would be a great feature, I have two implementations of a single opaque type I use currently already, one for Java and one for Ruby. But because your approach sounds very complex, maybe this issue could be further divided into single parts? Many languages have Java bindings already, so is there a way to implement opaque types in Java and use them using those bindings in e.g. ksv? Maybe such an approach could act as a least common denominator or PoC to provide only one implementation of an opaque type to different supported target runtimes?

[tempelmann]

I have a related need for my own tool "iBored", a cross-platform disk and file editor. It can, similarly to KS, read a file and process its contents based on a grammar, which iBored calls templates. The display from iBored is similar to what the Kaitai WebIDE shows: A structured view of the input file.

iBored cannot use the source files KS currently can generate. Instead, it needs a templates file, which is semantically similar to a ksy file, but with a very different syntax.

So, what I'd like to see is a way to automatically generate a "template" XML file for iBored from a ksy file.

The most direct option for this would be to write my own parser for ksy files, create an internal model of the ksy file's contents and then generate an iBored templates file from it. But I figure that the compiler's parser already does half of this work, so it would be helpful if I could just write code to output the internal ksy file's representation in my own format.

That code for outputting a templates file would still need to understand all the ksy's abilities, but by using an code-based API that's provided by the compiler's parser code, my output generator would not miss any fixes, improvements and additions to the ksy language to easily, I'd hope. Though, I could be wrong and doing my own parsing is actually safer, as then I'd notice any changes to the ksy format right away when I run into parsing errors. My fear is, however, that by writing my own parser, I might make mistakes by misinterpreting the ksy syntax.

One thing, however, that might help in general would be to define a computer-readable syntax description of the ksy format, so that parser-generators such as Bison, Yacc or ANTLR could be used. That might already be half the solution to this ticket.

So, I am not even sure if my goal can well be helped by this enhancement request. But I'll keep my eyes on this.

[koczkatamas]

One thing, however, that might help in general would be to define a computer-readable syntax description of the ksy format, so that parser-generators such as Bison, Yacc or ANTLR could be used. That might already be half the solution to this ticket.

KSY is a Yaml file, and Yaml parsing libraries are available to most platforms / languages. There were plans to create a schema documentation for Ksy, but there is no standard Yaml schema, so we created only a JSON Schema, but that was not updated lately.

[GreyCat]

@tempelmann To sum up our previous Twitter discussion, there are generally 2 ways to go with implementing KS into iBored:

1. Reuse current templates.xml infrastructure by writing a converter (i.e. compiler) from .ksy into templates.xml
2. Connect existing object exploration tool to some "engine" (as discussing in this issue) that would provide ready-made parsed object lists / values + offsets + size information.

Both have its own merits, but, after looking through current templates.xml state, I would say that method (1) would be probably harder to implement, due to templates.xml having less declarative features than .ksy specs. For example (correct me if I'm wrong), I believe there's no direct equivalent of the following KS things in iBored templates:

• instances — both parsed instances & value instances
• streams and substreams
• process
• enums
• things like _root and _parent to allow navigation between types ("structures" in iBored terminology) — see Accessing attributes in other types
• parametric types

On the other hand, iBored solves all that by inclusion of REALbasic's RbScript code. It's relatively easy to convert KS YAML into templates.xml formal syntax, but the real challenge would be to compile these more complex things into RbScript code.

This issue is probably relevant to method (2), if @tempelmann would consider implement it that way. This kind of short-circuits KS right into iBored core, bypassing templates.xml, working with some external server that returns stuff like a list of attributes on some particular path on a tree and its locations in the hex dump.

[tempelmann]

(This comment is iBored specific)

I like to keep using the templates format with iBored because it, too has some richer info that ksy does not have, which is mainly the ability to test whether a disk block matches a particular structure - iBored uses that to automatically detect matching structures, choosing the best when its option "Auto-templates" is checked. So, I imagine that the process for including ksy files with iBored would be a two-step process: First, convert the sky into a iBored template file, then manually added templates testing to the template file.

Or does the ksy format offer something similar? (I must admit I haven't look much at its syntax and abilities yet, because I only work with one specific ksy file currently, which doesn't use most of the above mentioned features).

[GreyCat]

@tempelmann Not yet, I guess. We do have some basic signature checking, but proper "validation" is not yet implemented. There are some proposals (for example, see #81), but nothing is set in stone yet.

[GreyCat]

Hey folks, I wanted to share some progress I had on this topic:

• I've recently learned more about OpenAPI and figured out it will be a perfect match for this project. Here's my initial take on OpenAPI spec for a protocol between visualizer <-> engine (visualizer is a client, engine is a server): https://github.com/kaitai-io/kaitai_struct_engine_openapi
• I've created pretty crude initial implementation of this engine in Python: https://github.com/kaitai-io/kaitai_struct_python_engine

If anybody cares to check it out:

python3 -m venv env
. env/bin/activate
python3 -m pip install -e ./
python3 -m kaitai_struct_python_engine

This starts server on http://127.0.0.1:5000

One can then either generate a client in any language from OpenAPI spec (e.g. by using https://openapi-generator.tech/), or just test it manually:

curl --verbose "$ENDPOINT/struct/load?local_file=tests/ksy/hello_world.ksy"
curl --verbose "$ENDPOINT/struct/load?local_file=tests/ksy/elf.ksy"
curl --verbose "$ENDPOINT/struct"
curl --verbose "$ENDPOINT/engine/load?local_file=tests/input/hello_world.bin&parser=hello_world"
curl --verbose "$ENDPOINT/engine/load?local_file=tests/input/write&parser=elf"
curl --verbose "$ENDPOINT/engine/explore?path="

There's also a bunch of tests available:

python3 -m pytest

[Mingun]

You can also explore the possibility offered by the Language Server Protocol (LSP) -- it is possible that you do not need to come up with your own protocol, but use a ready-made.

[GreyCat]

Not sure if I follow? Language server is basically about adding support for better editing of language content in IDE like vscode — it's about completion, navigation, integration with build tools and debuggers.

This issue is different — it's about decoupled "engine" which keeps state of parsing in memory and performs the parsing itself by executing code vs separately ran "visualizer" which explores the tree and draws it.

[Mingun]

Language server is about parsing the something and answer queries about that. Maybe some of those queries are suitable for your idea. About this my proposal is to investigate whether it is possible to use a ready-made protocol.

Visualizer then can be LSP client which issues standard queries to the LSP server.