Radical .ksy format changes proposal

[GreyCat]

With some time passing and some experience gained, I see that we could review some of the original design decisions for .ksy format. In this topic, I'll try to gather a pretty radical ideas (=breaking current format specs) and try to tell the rationale I see behind it.

Given that transition process might be pretty painful (if it would happen), at the very least, we should do a pack of such changes in one shot (probably while going from v0 => v1) and we should provide automated converters, "ksy level" specs and switches, backward compatibility support in the compiler, etc.

I'll post each proposal as separate comment, so it could be referenced, discussed, voted for/against, etc.

[GreyCat]

Get rid of meta/id

Instead of regular boilerplate:

meta:
  id: foo
seq:
  - id: one
    type: u1

... let's just start a file with a map of (type name => type spec) that goes right now inside a types tag. This would yield us:

foo:
  seq:
    - id: one
      type: u1

Pros

• Much more consistent. No magic meta/id at top level, which must be artificially banned on non-top levels.
• One can specify many top-level formats in .ksy, without resorting to nesting, which might be unnecessary.
• One can concatenate several .ksy files into one with just cat a.ksy b.ksy c.ksy >total.ksy, compile it, and it would still be fine and valid .ksy.

Cons

• Extra level of indent.
• In practice, meta on top level type would be required almost always anyway. Almost any non-trivial format would specify meta/endian, any decent format spec should have application and/or file-extension, etc. Most likely, "minimal compiler version required " or "ksy level required" would go into meta as well.
• In practice, nesting types would still be a good idea (as it will automatically spread meta/endian to all subtypes, and it will hide format implementation details from language's global namespace).

[GreyCat]

u4 → uint32, etc

I don't personally like the idea, but after explaining KS to a few dozen people already, I must admit that given that the majority of people come from C background, they find it much easier to understand uint8 or int32.

Pros

• More understandable by average user (C, C++, JavaScript, C# use such notation; Perl, Ruby and Python refer to them )
• Might play good with bit specification syntax (i.e. uint3 for 3-bit integer).
• Might not break compatiblity completely (technically it's possible to support both notations at the same time).

Cons

• Longer.
• These are the things that people keep typing a lot, so it would definitely bring some heavy disagreement to existing users.
• Floats would thus become float32 and float64, which is definitely not C standard in any way.
• Allowing both notations at once would bring total chaos.

[athre0z]

Personally, the only thing about the types I found confusing was that they use byte granularity. How about keeping the short type prefixes, just changing byte to bit notation? Types like u32 and s32 should be well known to most C programmers (e.g. the Linux kernel uses them) and also are the standard types in some modern languages like Rust.

This would break compatibility though, but as the project is pretty fresh, I guess that's not too much of a big deal.

[GreyCat]

Types like u32 and s32 should be well known to most C programmers

That's definitely not a good idea: we'll have a bad clash with u8 meaning either 8 bits of 8 bytes (both are valid), thus wreaking total chaos instead of clean error message. If we go that way, probably U8-U16-U32-U64 is the way, or something like that.

well known to most C programmers (e.g. the Linux kernel uses them) and also are the standard types in some modern languages like Rust.

True enough, but u1-u2-u4 notation also came one not from the thin air, but has some use. For example, Java uses it for specs, Adobe uses it for Flash (although inconsistently), Open Watcom uses it, various emulators like zsnes, etc, etc.

It's hard to do solid estimates, but I might guess that u8-u16-u32 is about just as confusing as u1-u2-u4. Although probably it's a good idea to do some sort of general review of what modern programming languages use. For example:

• Go uses int32-uint32-float32.
• Rust uses i32-u32-f32.
• D uses int-uint-float (and short for 16-bit, long for 64-bit, etc).
• Scala follows mostly the same suit (mostly following Java, though) and lacks unsigneds.
• Swift has Int32, UInt32, Float (and Double for 64-bit float).
• C/C++ use int32_t, uint32_t and float (+double).

[koczkatamas]

Added ksy modularization proposal for discussion here: https://github.com/kaitai-io/kaitai_struct/issues/71

[Mingun]

Get rid of ks-opaque-types

Instead all opaque types must be imported explicitly, for example, under key meta/external-types, or /external-types:

meta:
  external-types:
    - my_opaque_type1
    - my_opaque_type2
    - ...

Pros

• Explicit is better, then implicit
• Compiler can protect from misprints
• More informative

Cons

• More verbose (but that by design)

[Mingun]

Rework repeat*, size* and terminator family keys

Proposal:

- id: field
  # shortcut:
  # repeat: <int-expression>
  repeat:
    # Only one key allowed
    count: <int-expression>
    until: <bool-expression>
    while: <bool-expression>
    to-eos: true
  # shortcut:
  # size: <int-expression>
  size:
    # Only one of two following keys allowed
    value: <int-expression>
    to-eos: true
    # shortcut:
    # terminator: <byte-value>
    terminator:
      value: <byte-value>
      consume: true
      include: false
      mandatory: true # replaces poor named `eos-error`

Pros

• New structure reflects scope of keys
• More descriptive names, such as to-eos, mandatory, count
• Frequently used settings requires only one line (shortcuts)

Cons

• More indentations

[Mingun]

Make _parent variable a tuple/array with all parents

So getting parent of parent would _parents.1 instead of _parent._parent. Regular _parent becomes _parents.0. Also if it is more preferrable, array indexing style can be choosed: _parents[0], _parents[1], etc.

Pros

• Long chains of parents gets shorter

Cons

• It is not obvious whether indexing is from one or from zero

[KOLANICH]

Instead all opaque types must be imported explicitly, for example, under key meta/external-types, or /external-types:

314