Add support for bin protocol parsing (kaitai)

[sirdarckcat]

It would be easy to create something that parses and let's the user visualize the binary format presented like ide.kaitai.io

The harder part would be to make it useful and editable. Binary protocols often have length and checksums all over the place, which aren't obviously editable even with kaitai.

But even if this wasn't possible to edit easily, it might be something nice to present to users for binary formats like protobuffs as well as for some media elements.

This is more useful probably for responses than for requests, unless we add a mime envelope parser, as file uploads are often encoded like that. That said, it does happen that some websites generate binary protobuff on the client.

[sirdarckcat]

https://github.com/kaitai-io/kaitai_struct_typescript_runtime

[sirdarckcat]

https://doc.kaitai.io/lang_javascript.html might be better, although looking at the API, it seems like the offsets arent exposed in the API, only the parsed objects.. this means it might be hard to associate the offset at which a specific field is located.

https://github.com/kaitai-io/kaitai_struct_webide/tree/master/src/worker

[sirdarckcat]

hmm object offsets seem to be exposed via _byteOffset in a somewhat hacky way https://github.com/kaitai-io/kaitai_struct_javascript_runtime/blob/master/KaitaiStream.js

but even then

  function Elf(_io, _parent, _root) {
    this._io = _io;
    this._parent = _parent;
    this._root = _root || this;

    this._read();
  }
  Elf.prototype._read = function() {
    this.magic = this._io.readBytes(4);

And the primitives byte offsets are exposed via Uint8Array native byteOffset property.

[sirdarckcat]

so, it might be possible to just iterate over all defined properties of an object, get their byteOffset (and byteLength) if they are Array views, and get _byteOffset otherwise.

[sirdarckcat]

from the example in http://doc.kaitai.io/lang_javascript.html:

> parsedElf._io.byteOffset
0
> parsedElf.magic
Uint8Array [ 127, 69, 76, 70 ]
> parsedElf.magic.byteOffset
0
> parsedElf.header.flags
Uint8Array [ 0, 0, 0, 0 ]
> parsedElf.header.flags.byteOffset
48

seems to work at least for things that are arrays, but it does not seem to work for things that are returned as primitives

> parsedElf.header.machine._io
undefined
> parsedElf.header.machine.byteOffset
undefined

it's unclear how the webide gets this information

[sirdarckcat]

this is exposed by the compiler it seems

            this.ksySchema = <KsySchema.IKsyFile>YAML.parse(srcYaml);
            this.ksyTypes = SchemaUtils.collectKsyTypes(this.ksySchema);

[sirdarckcat]

so, a possibly easier solution to this would be to provide a fake implementation of KaitaiStream that stores the information of offsets for all the values it returns.

[sirdarckcat]

So, I would like to recover the parsing offsets information (eg, where each value came from, so I can do something similar to the ide.kaitai.io hex editor, and know where each offset corresponds to), this isn't exposed on the KaiaiStream API, but seems possible to recover by providing a fake implementation of KaitaiStream that stores the information of offsets for all the values it returns in a Map.

Example:

KaitaiStream.prototype.readS1 = function() {
  this.ensureBytesLeft(1);
  var v = this._dataView.getInt8(this.pos);
  this.pos += 1;
  return v;
};

would be:

KaitaiStream.prototype.readS1 = function() {
  this.ensureBytesLeft(1);
  var v = this._dataView.getInt8(this.pos);
  this.pos += 1;
  return this._addToMap(v); // <<-- this is new
};

with

KaitaiStream.prototype._map = new Map; // (this has to be set in the constructor)
KaitaiStream.prototype._last = 0;
KaitaiStream.prototype._addToMap = (v) => {
  const obj = new v.constructor(v);
  this._map.set(obj, {index: this._map.size, start: this._last, end: this.pos});
  this._last = this.pos;
  return obj;
};

then when reading the values, I can just search them on the map.

[sirdarckcat]

@koczkatamas (hopefully this works ^.^) would https://github.com/google/tamperchrome/issues/63#issuecomment-753526073 work?

[koczkatamas]

@koczkatamas (hopefully this works ^.^) would #63 (comment) work?

KS has a somewhat hidden 'debug' mode which does exactly what you need: adds a _debug field into objects and stores the start, end and so-called ioOffset of the fields.

If you generate the parser code with the WebIDE, then use the JS code (debug) tab here:

Otherwise if you use the KS JS compiler, then you can set the debug property of the compile method to true to generate debug code instead of a non-debug one: https://github.com/kaitai-io/kaitai_struct_compiler/tree/master/js#basic-usage-of-compile-method

I hope this is what you need :)

[sirdarckcat]

woohoo! thanks =D