Reconsider how file formats are described here?

Luashine commented 2 years ago

The way you intended the formats to be described is one file per format version. That'd work well, if the file formats were radically different, but they're not. Looking at w3i (Info), the file format evolved incrementally, where each version only added new fields in between, without major changes.

I really liked how Hodor described the formats here (in Russian). Open war3map.w3i as an example, additional fields of newer versions are conveniently described within IF VERSION > 28 tags, which makes multi-version support trivial when writing a parser.

Bringing everything to one file (as long as the formats stay roughly the same) has the advantage that all changes don't affect 2+ files. Further, while writing a parser it's practically zero effort to support multiple formats. IF-ELSE-PROFIT!

I'd rewrite the format descriptions later and make a PR if you agree. I would probably follow Hodor's approach too, its readability is miles ahead of this: https://github.com/stijnherfst/HiveWE/wiki/war3map.wtg-Triggers

Further resources:

https://xgm.guru/p/wc3/w3-file-format by Hodor from above
https://www.hiveworkshop.com/threads/parsing-metadata-from-w3m-w3x-w3n.322007/
@Drake53's .NET library: https://github.com/Drake53/War3Net/tree/master/src/War3Net.Build.Core (has EnemyPlayerPriorities described, last unknown thing in current Info)

Luashine commented 2 years ago

Here's Hodor's article from XGM as a screenshot:

hodor-example

ChiefOfGxBxL commented 2 years ago

Thanks, I'll review the different formats and consider it. When I considered this in the past I was concerned about the documents becoming messy with exceptions, and didn't consider the redundancy in the file versions to be a big deal. I'll think about it again.

WaterKnight commented 1 year ago

Consider adding formal Kaitai Struct specifications https://kaitai.io/ for more structural exactness, parser generation and at some point also writer generation https://github.com/kaitai-io/kaitai_struct/issues/27 https://doc.kaitai.io/serialization.html.

Web IDE: ide.kaitai.io

Here is a .ksy file for .w3i files I have begun to write:

meta:
  id: wc3_w3i
  file-extension: w3i
  endian: le
seq:
  - id: version
    type: u4
  - id: saves_amount
    type: u4
    if: version >= 16
  - id: editor_version
    type: u4
    if: version >= 16
  - id: game_version
    type: game_version
    if: version >= 27
  - id: map_name
    type: w3str
  - id: author
    type: w3str
  - id: recommended_players
    type: w3str
    if: version >= 8
  - id: unknown_v0_to_v3
    type: unknown_v0_to_v3
    if: version <= 3
  - id: unknown_v4_to_v8
    type: unknown_v4_to_v8
    if: version > 3 and version <= 8
  - id: unknown_string
    type: w3str
  - id: camera_bounds
    type: camera_bounds
  - id: margins
    type: margins
    if: version >= 14
  - id: map_width
    type: u4
    if: version >= 1
  - id: map_height
    type: u4
    if: version >= 1
  - id: unknown_v3_int1
    type: u4
    if: version >= 2 and version <= 8
  - id: flags
    type: flags
    if: version >= 2
  - id: tileset
    type: w3char
    if: version >= 8
  - id: loading_screen
    type: loading_screen
  - id: game_data_set
    type: u4
    if: version >= 17
  - id: unknown_path
    type: w3str
    if: version <= 17
  - id: prologue_screen
    type: prologue_screen
    if: version >= 13
  - id: fog
    type: fog
    if: version >= 19
  - id: global_weather_id
    type: u4
    if: version >= 21
  - id: sound_environment
    type: w3str
    if: version >= 22
  - id: light_environment
    type: w3char
    if: version >= 23
  - id: water_color
    type: water_color
    if: version >= 25
  - id: script_languageso
    type: u4
    if: version >= 28
  - id: supported_graphics_modes
    type: u4
    if: version >= 29
  - id: game_data_version
    type: u4
    if: version >= 30
  - id: players_chunk
    type: players_chunk
  - id: forces_chunk
    type: forces_chunk
  - id: upgrades_chunk
    type: upgrades_chunk
    if: version >= 6
  - id: techs_chunk
    type: techs_chunk
    if: version >= 7
  - id: random_unit_tables_chunk
    type: random_unit_tables_chunk
    if: version >= 12
  - id: random_item_tables_chunk
    type: random_item_tables_chunk
    if: version >= 24
  - id: script_language2
    type: u4
    if: version == 26 or version == 27

types:
  w3str:
    seq:
      - id: value
        type: str
        terminator: 0
        encoding: UTF-8
  w3char:
    seq:
      - id: value
        type: str
        encoding: UTF-8
        size: 1
  w3id:
    seq:
      - id: value
        type: str
        encoding: UTF-8
        size: 4
  point_2d:
    seq:
      - id: x
        type: f4
      - id: y
        type: f4
  game_version:
    seq:
      - id: major
        type: u4
      - id: minor
        type: u4
      - id: revision
        type: u4
      - id: build
        type: u4
  unknown_v0_to_v3:
    seq:
      - id: unknown_v0_int1
        type: u4
      - id: unknown_v0_int2
        type: u4
  unknown_v4_to_v8:
    seq:
      - id: unknown_v4_int1
        type: u4
      - id: unknown_v4_int2
        type: u4
      - id: unknown_v4_int3
        type: u4
      - id: unknown_v4_float1
        type: f4
      - id: unknown_v4_float2
        type: f4
      - id: unknown_v4_float3
        type: f4
  camera_bounds:
    seq:
      - id: camera_bound_left
        type: point_2d
      - id: camera_bound_bottom
        type: point_2d
      - id: camera_bound_right
        type: point_2d
      - id: camera_bound_top
        type: point_2d
  margins:
    seq:
    - id: left
      type: u4
    - id: right
      type: u4
    - id: bottom
      type: u4
    - id: top
      type: u4
  flags:
    seq:
      - id: hide_minimap_on_preview_screens
        type: b1
      - id: change_ally_priorities
        type: b1
      - id: melee
        type: b1
      - id: non_default_tileset_map_size_large_never_been_reduced_to_medium
        type: b1
      - id: unexplored_areas_partially_visible
        type: b1
      - id: fixed_player_parameters_for_custom_teams
        type: b1
      - id: use_custom_teams
        type: b1
      - id: use_custom_techs
        type: b1
      - id: use_custom_abilities
        type: b1
      - id: use_custom_upgrades
        type: b1
      - id: map_properties_menu_opened_at_least_once
        type: b1
      - id: show_water_waves_on_cliff_shores
        type: b1
      - id: show_water_waves_on_rolling_shores
        type: b1
      - id: use_terrain_fog
        type: b1
      - id: tft_required
        type: b1
      - id: item_classification
        type: b1
      - id: custom_water_tint_color
        type: b1
      - id: rest
        type: b1
        repeat: expr
        repeat-expr: 15
  loading_screen:
    seq:
      - id: loading_screen_index
        type: u4
        if: _root.version >= 17
      - id: custom_loading_screen_path
        type: w3str
        if: _root.version >= 10 and _root.version != 18 and _root.version != 19
      - id: loading_screen_text
        type: w3str
        if: _root.version >= 10
      - id: loading_screen_title
        type: w3str
        if: _root.version >= 11
      - id: loading_screen_subtitle
        type: w3str
        if: _root.version >= 11
  prologue_screen:
    seq:
      - id: prologue_screen_path
        type: w3str
        if: _root.version >= 18
      - id: prologue_screen_text
        type: w3str
      - id: prologue_screen_title
        type: w3str
      - id: prologue_screen_subtitle
        type: w3str
  fog:
    seq:
      - id: fog_type
        type: u4
      - id: fog_z_start
        type: f4
      - id: fog_z_end
        type: f4
      - id: fog_density
        type: f4
      - id: fog_color_red
        type: u1
      - id: fog_color_green
        type: u1
      - id: fog_color_blue
        type: u1
      - id: fog_color_alpha
        type: u1
  water_color:
    seq:
      - id: red
        type: u1
      - id: green
        type: u1
      - id: blue
        type: u1
      - id: alpha
        type: u1
  players_chunk:
    seq:
      - id: amount
        type: u4
      - id: player
        type: player
        repeat: expr
        repeat-expr: amount
  player:
    seq:
      - id: num
        type: u4
      - id: controller
        type: u4
      - id: race
        type: u4
      - id: fixed_position
        type: u4
      - id: nme
        type: w3str
      - id: position
        type: point_2d
      - id: ally_low_priority
        type: u4
        if: _root.version >= 5
      - id: ally_high_priority
        type: u4
        if: _root.version >= 5
      - id: enemy_low_priority
        type: u4
        if: _root.version >= 31
      - id: enemy_high_priority
        type: u4
        if: _root.version >= 31
  forces_chunk:
    seq:
      - id: amount
        type: u4
      - id: force
        type: force
        repeat: expr
        repeat-expr: amount
  force:
    seq:
      - id: flags
        type: u4
      - id: players
        type: u4
      - id: name
        type: w3str
  upgrades_chunk:
    seq:
      - id: amount
        type: u4
      - id: upgrade
        type: upgrade
        repeat: expr
        repeat-expr: amount
  upgrade:
    seq:
      - id: players
        type: u4
      - id: id
        type: w3id
      - id: level
        type: u4
      - id: availability
        type: u4
  techs_chunk:
    seq:
      - id: amount
        type: u4
      - id: tech
        type: tech
        repeat: expr
        repeat-expr: amount
  tech:
    seq:
      - id: players
        type: u4
      - id: id
        type: w3id
  random_unit_tables_chunk:
    seq:
      - id: amount
        type: u4
      - id: table
        type: random_unit_table
        repeat: expr
        repeat-expr: amount
  random_unit_table:
    seq:
      - id: num
        type: u4
      - id: name
        type: w3str
      - id: columns_amount
        type: u4
      - id: column_type
        type: u4
        repeat: expr
        repeat-expr: columns_amount
      - id: rows_amount
        type: u4
      - id: row
        type: random_unit_table_row
        repeat: expr
        repeat-expr: rows_amount
  random_unit_table_row:
    seq:
      - id: chance
        type: u4
        repeat: expr
        repeat-expr: _parent.columns_amount
  random_item_tables_chunk:
    seq:
      - id: amount
        type: u4
      - id: table
        type: random_item_table
        repeat: expr
        repeat-expr: amount
  random_item_table:
    seq:
      - id: num
        type: u4
      - id: name
        type: w3str
      - id: sets_amount
        type: u4
      - id: set
        type: item_set
        repeat: expr
        repeat-expr: sets_amount
  item_set:
    seq:
      - id: items_amount
        type: u4
      - id: item
        type: item
        repeat: expr
        repeat-expr: items_amount
  item:
    seq:
      - id: chance
        type: u4
      - id: id
        type: w3id

Luashine commented 1 year ago

:cry: https://github.com/Luashine/wc3-file-formats

:cry: https://github.com/Luashine/wc3-file-formats/blob/b7f578771933a9f64c405ea9d7355513f80d6a51/w3i-finished-notpushed/WIP-stable.ksy how much time did you spend?

Edit: I like your ksy style

WaterKnight commented 1 year ago

@Luashine ~4 hours, using https://github.com/ChiefOfGxBxL/WC3MapSpecification/blob/36c1abfe1d8a1d420e7be2b8d376f601924a3cb5/Info/0-31.md and wc3libs. It was an excuse to look into Kaitai Struct finally. But it's only a stub and was tested only on a single .w3i file. For example, all the string length limits are missing and enum mappings.

ChiefOfGxBxL commented 1 year ago

Very nice - I'll read up more about Kaitai on their website. Since Luashine's wc3-file-formats repo is mentioned here, I'll reach out to various people on The Hive (Discord) to see if we can consolidate efforts under a Hive GitHub organization. There's a bunch of duplication out there that we might be able to consolidate.

WaterKnight commented 1 year ago

I often search by the file name or extension (if that is unique) rather than "Units" or "Info". "Units" could also mean "war3map.w3u" or the unit slks. And I hope we will have more custom formats in the future. As folder organization, it is okay, but could the files be renamed or tagged otherwise, so it is easier to find them? Thus, you can also more easily see what's still missing.

ChiefOfGxBxL / WC3MapSpecification

Reconsider how file formats are described here? #3