search

kaitai-io / kaitai_struct

Kaitai Struct: declarative language to generate binary data parsers in C++ / C# / Go / Java / JavaScript / Lua / Nim / Perl / PHP / Python / Ruby
https://kaitai.io
3.96k stars 192 forks source link

Creating a counter #476

Open joshewilliams opened 5 years ago

joshewilliams commented 5 years ago

I was wondering if there was a good way to create a counter. I'm not sure of a good way to describe this, so I'll just go with the example I'm trying to work on. I'm attempting to create a file format parser for the U3D File Format.

Specifically, I'm attempting to handle the Stay or Move field described in 9.6.1.3.4.7 (page 72) which is in the resolution_update type. The Stay or Move field is repeated based on the number of times that the face_orientation in new_face_position_info is 0x01.

Following is my template in its current state, omitting all but the relevant parts:

  resolution_update:
    seq:
      - id: split_position_index
        type: u4
      - id: new_diffuse_color_info
        type: new_diffuse_color_info
      - id: new_specular_color_info
        type: new_specular_color_info
      - id: new_texture_coord_info
        type: new_texture_coord_info
      - id: new_face_count
        type: u4
      - id: new_face_position_info
        type: new_face_position_info
        repeat: expr
        repeat-expr: new_face_count
      - id: stay_or_move
        type: u1
  new_face_position_info:
    seq:
      - id: shading_id
        type: u4
      - id: face_orientation
        type: u1
        enum: face_orientation
      - id: third_position_type
        type: u1
        enum: third_position_type
      - id: local_third_position_index
        type: u4
        if: third_position_type == third_position_type::local_third_position_index
      - id: global_third_position_index
        type: u4
        if: third_position_type == third_position_type::global_third_position_index

I can't think of a good way to do this. If there isn't one, can some sort of counter to Kaitai Struct?

GreyCat commented 5 years ago

Unfortunately, right now there's no good answer to this class of problems in KS.

"Counter", by definition, is imperative, i.e. it relies on a certain way to process (in this case, sequentially parse) the stream.

KS strives to be declarative, i.e. not rely on a particular implementation, but rather describe the essence of the data itself and relationships between data elements. Granted, current implementations that ksc generates is more or less sequential and eager, but it does not mean that other possible implementations would not be so straightforward. For example, we can't rely on the process being parsing (i.e. reading), as opposed to serialization (i.e. writing), and we can't rely on it being sequential and eager (i.e. not lazy).

Probably the best way to solve this class of problems in KS would be to introduce some collection operations that one can invoke in expression language. Given that you state that:

The Stay or Move field is repeated based on the number of times that the face_orientation in new_face_position_info is 0x01.

... we could do it like that:

      - id: stay_or_move
        type: u1
        repeat: expr
        repeat-expr: 'new_face_position_info.count { |x| x.face_orientation == 0x01 }'

where collection.count() being a method which takes one lambda expression / anonymous function (that accepts one collection element and is expected to return boolean), runs it for every element of the collection (passing that element inside) and then counts the number of times it returned true.

Unfortunately, this is not implemented now :(

joshewilliams commented 5 years ago

Thanks for the quick response! Looks like I might be out of luck with this particular format for now, or I can just ignore the rest of that data stream for the moment. Fortunately, that isn't the only file format I need to work with, so I'll just move on to the others for now.

GreyCat commented 5 years ago

Well, if all else fails, you can always implement that particular type imperatively in your language of choice, i.e. see opaque types.

Maxzor commented 4 years ago

Any update?

Maxzor commented 4 years ago

Thanks to @generalmimon on gitter.im & https://github.com/kaitai-io/kaitai_struct_formats/pull/265#discussion_r377966739 , here is a wip ksy file for git packfile index (.git/objects/pack/*idx) that counts : https://github.com/kaitai-io/kaitai_struct_formats/issues/323

KOLANICH commented 4 years ago

here is a ksy file for git packfile index

Could you create a PR into KSF?

Maxzor commented 4 years ago

Will once packfile is done