Abilities

[acstanton515]

initial foundation for adding abilities data

[robertvanhoesel]

(copied over from Discord for completeness)

few important facts that drive a few important decisions: source/abilities and source/info/buff_info are very much lacking the details that are present in units, buildings, techs, and upgrades some abilities live in both source paths, some live in only one abilities don't really have classes there are about 40 abilities of note +/- 10 abilities interact with weapons (specific weapons) abilities interact with techs (unlocks)

decisions may delete some props of Items in Ability if extend Items into Ability, or have to better organize the progression of the main func in parse.ts may hardcode all paths of abilities in config.ts to pick between source/abilities path and source/info/buff_info path may move modifiers of ability unlocking techs to ability, or could leave these out of abilities may have to hardcode most properties of abils in a similar manner to technologies.ts or perhaps workarounds.ts (leaning on like technologies.ts except most abilities dont have the same extended info into locale/en/cardinal.en.ucs)

Looks really good already. I agree the format doesn't fit so wel with the current generic Items. I may be inclined to say we should create a different model and concept all together. Before we do so I think it is wise to see what data properties are available from the game files and which would possibly need to be added by hand.

As for discovery I think we should start from units/buildings, and unpack the possibly related abilities from there. Each unit and building will have the following data properties that we could inspect.

attrib/instances/ebps/races/abbasid/buildings/building_house_of_wisdom_control_abb.xml

<template_reference name="exts" value="ebpextensions\ability_ext" List.ItemID="-335360918" List.ParentItemID="-335360918">
    <list name="abilities" overrideParent="True">
        <instance_reference name="ability" value="abilities\always_on_abilities\abbasid\golden_age\golden_age_tier_1" List.ItemID="-1286633340" />
        <instance_reference name="ability" value="abilities\always_on_abilities\abbasid\golden_age\golden_age_tier_2" overrideParent="True" List.ItemID="390938176" />
        <instance_reference name="ability" value="abilities\always_on_abilities\abbasid\golden_age\golden_age_tier_3" overrideParent="True" List.ItemID="1521192705" />
        <instance_reference name="ability" value="abilities\always_on_abilities\abbasid\passives\golden_age_bonus_1_abb" List.ItemID="-1959297083" />
        <instance_reference name="ability" value="abilities\modal_abilities\abbasid\golden_age_passive_abb" List.ItemID="1052642022" />
        <instance_reference name="ability" value="abilities\always_on_abilities\abbasid\passives\how_influence_abb" overrideParent="True" List.ItemID="-953833507" />
    </list>
</template_reference>

The filenames in this list is normalized with inherited properties to ebpExts.ability_ext in parse.ts. Which could be added as references on the item, and then deduplicated in a step like https://github.com/aoe4world/data/pull/18/files#diff-f65287e281bf83f4283990f21c513df8df96655d4ece811024d8a8652a2131e3R79-R80 in run.ts

From there we can fully parse the contents of all the abilities.

The nice thing here is that in UIs like Explorer, you can work from the item/building buttons and passive abilities that may be available and look them up.

[acstanton515]

As for discovery I think we should start from units/buildings, and unpack the possibly related abilities from there. Each unit and building will have the following data properties that we could inspect.

I started with this early on, and while it does grab many abilities; some are non-interesting (thus filtered out civ_core), missing, or wrong; I have some of the code from that experiment. There are approximately 350 files in abilities.

parse.ts and items.ts

diff --git a/src/attrib/parse.ts b/src/attrib/parse.ts
index f3f479f..fcd9b39 100644
--- a/src/attrib/parse.ts
+++ b/src/attrib/parse.ts
@@ -124,6 +124,8 @@ export async function parseItemFromAttribFile(file: string, data: any, civ: civC
           }
         }

+      const abilitiesExts = findExt(data, "squadexts", "sbpextensions/squad_ability_ext");
+
       const unit: Unit = {
         ...item,
         type: "unit",
@@ -133,6 +135,7 @@ export async function parseItemFromAttribFile(file: string, data: any, civ: civC
         sight: parseSight(ebpExts?.sight_ext),
         movement: parseMovement(ebpExts?.moving_ext),
         garrison: parseGarrison(ebpExts?.hold_ext),
+        abilities: parseAbilities(abilitiesExts),
       };

       return unit;
@@ -304,6 +307,15 @@ function parseUnique(ui_ext: any) {
   return ui_ext.is_unique_to_race || ui_ext.is_unique || ["UniqueBuildingUpgradeDataTemplate", "BuildingImprovedUpgradeDataTemplate"].includes(ui_ext.tooltip_data_template);
 }

+function parseAbilities(abilities_ext: any) {
+  return abilities_ext?.abilities
+    ? {
+      type: "ability",
+      abilities: abilities_ext?.abilities?.map((x) => (typeof x == "string" ? x : x.ability)).filter((x) => !x.includes("civ_core")) ?? [],
+    }
+    : undefined;
+}
+
 function parseInfluences(ui_ext: any) {
   if (ui_ext.ui_extra_infos)
     return ui_ext.ui_extra_infos?.reduce((inf, x) => {
diff --git a/src/types/items.ts b/src/types/items.ts
index 0f488fd..ee3c89f 100644
--- a/src/types/items.ts
+++ b/src/types/items.ts
@@ -84,6 +84,7 @@ export interface PhysicalItem extends Item {

 export interface Unit extends PhysicalItem {
   type: "unit";
+  abilities?: string;
   movement: {
     speed: number;
   };

Results:

..longbowman-2...
+      "abilities": {
+        "type": "ability",
+        "abilities": [
+          "abilities/toggle_abilities/english/longbow_set_up_camp",
+          "abilities/timed_abilities/english/longbow_rate_of_fire_ability",
+          "abilities/modal_abilities/core/military_neutralize_holy_site"
+        ]
       }

...royal-knight-2...
+      "abilities": {
+        "type": "ability",
+        "abilities": [
+          "abilities/always_on_abilities/french/lancer_charge_melee_boost_fre",
+          "abilities/modal_abilities/core/military_neutralize_holy_site",
+          "abilities/always_on_abilities/core/military_charge"
+        ]
       }

...ghulam-3...
+      },
+      "abilities": {
+        "type": "ability",
+        "abilities": [
+          "abilities/modal_abilities/core/military_neutralize_holy_site",
+          "abilities/always_on_abilities/core/military_charge"
+        ]
       }

...camel-rider-3...
+      "abilities": {
+        "type": "ability",
+        "abilities": [
+          "abilities/modal_abilities/core/military_neutralize_holy_site"
+        ]
       }

...mehter-2...
+      "abilities": {
+        "type": "ability",
+        "abilities": [
+          "abilities/modal_abilities/core/military_neutralize_holy_site",
+          "abilities/always_on_abilities/core/military_charge"
+        ]
       }

We can also implement discovery in run.ts which I did in my next approach, before landing where I am now on hard-coded discovery:

run.ts

function findAbilities(data: any) {
  if (data?.extensions?.find((x) => x.exts === "ebpextensions/ability_ext"))
    return data?.extensions?.find((x) => x.exts === "ebpextensions/ability_ext")?.abilities?.flatMap((u) => u.ability ?? u) ?? [];
  if (data?.extensions?.find((x) => x.exts === "sbpextensions/squad_ability_ext"))
    return data?.extensions?.find((x) => x.exts === "sbpextensions/squad_ability_ext")?.abilities?.flatMap((u) => u.ability ?? u) ?? [];
}

Before we do so I think it is wise to see what data properties are available from the game files and which would possibly need to be added by hand.

And this is where I feel the files are useful for basically name, description, and icon. Anything else is hit or miss. I think I found a few abilities that implement help_text_formatter. recharge_time and range may be fairly accurate to capture in source/abilities.

The ToV buff isnt even in source/abilities (the influence aspect of it is, but I only find ToV details in info/buff_info).

I started to move some of my notes on what data is available in these files to this Google Sheets, but have not fully captured the details.

In conclusion,

I am open to doing it any of these ways, but currently favor hard coding 40-50 abilities.

We may use parseAbilities for certain abilities that are applied to many units like charges and spearwall and actually make that a prop of Units (i.e. charge: true or spearwall: true)

And my apologies, I am kind of focused on abilities for units for phase 1 but definitely keeping in mind how it would extend to buildings.

[acstanton515]

The nice thing here is that in UIs like Explorer, you can work from the item/building buttons and passive abilities that may be available and look them up.

I'd imagine you could do this the same way you do with technologies in Explorer, by looking for matches in the modifier of the ability. I think the func in Explorer is modifierMatches

[acstanton515]

@robertvanhoesel - in todays commit, I have:

• kept Ability extends Item by moving some of the properties to downstream interfaces and re-ordered parse.ts to handle those properties similarly downstream (this results in lots of changes to data files as the properties appear in a different order in the JSON, and I left those out of the commit for now). I went this route as Item does truly have many/most properties we should adopt into Ability.
• pulled the first set of data properties from the initial set of hardcoded source/attrib/instances/abilities files
• retrofitted technologyModifiers to be able to apply to abilities as well (there is a section in technology in parse.ts where you attempt to pull the properties from float_properties, but I dont think this is actually used, is it?)

You may like what you see in the abilities/all.json now.

Let me know what you don't like. ~ A

[acstanton515]

@robertvanhoesel

i returned class handling to the way it was so abilities does not impact files of existing data sources. keeping on going, my current struggle is with techs that unlock abilities.

how do you want to handle techs that unlock abilities in terms of identification? They may have the same baseId and/or id and throw error if not done differently.

Error: Duplicate item id arrow-volley-4 in abilities/timed_abilities/english/longbow_rate_of_fire_ability conflicts with upgrade_ranged_longbow_arrow_volley_eng

I could implement overrideAge workaround for ability to properly reflect the age it is available when it is being unlocked, or create some dependency between techs that unlock abilities, but so far, there is no precedence of relationship of one item to the next.

Again, abilities are lacking detail as the requirements for arrow volley ability only mention the tech.

                <list name="requirements">
                        <template_reference name="required" value="requirements\required_player_upgrade" List.ItemID="-1228156972">
                                <enum name="reason" value="usage_and_display" />
                                <locstring name="ui_name" value="0" />
                                <bool name="is_present" value="True" />
                                <int name="max_completed" value="100" />
                                <int name="min_completed" value="1" />
                                <instance_reference name="upgrade_name" value="upgrade\races\english\research\upgrade_ranged_longbow_arrow_volley_eng" />
                                <bool name="include_completed" value="True" />
                                <bool name="include_queued" value="False" />
                                <group name="include_pbg_parenting">
                                        <bool name="include_child_pbgs" value="False" />
                                        <bool name="include_parent_pbgs" value="False" />
                                </group>
                        </template_reference>
                </list>

[acstanton515]

Appears to be no ill will in refining the error check by type:

    if (items.has(item.id)) {
      if (items.get(item.id)!.type == item.type) {
        throw new Error(`Duplicate item id ${item.id} in ${file} conflicts with ${items.get(item.id)!.attribName}`);
      }
    }

Then keep separate lists of modifiers:

import { technologyModifiers, abilityModifiers } from "./modifiers";

[acstanton515]

Abilities are starting to get added quickly now that many issues are sorted.

[acstanton515]

@robertvanhoesel are you comfortable with the file name change for technologies.ts to modifiers.ts? Would you be able to resolve the conflict when you are ready to pull?

[robertvanhoesel]

@robertvanhoesel are you comfortable with the file name change for technologies.ts to modifiers.ts? Would you be able to resolve the conflict when you are ready to pull?

If it's not too much, maybe best to create a folder 'modifiers' with index that merges abilities.ts and technologies.ts? I agree it needs a bit of restructure, just hoping to make it easy for ourselves to navigte

[acstanton515]

do we want to delete classes property of abilities; currently all empty:

data (abilities) $ grep classes abilities/all.json "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [], "classes": [],

[acstanton515]

@robertvanhoesel are you comfortable with the file name change for technologies.ts to modifiers.ts? Would you be able to resolve the conflict when you are ready to pull?

If it's not too much, maybe best to create a folder 'modifiers' with index that merges abilities.ts and technologies.ts? I agree it needs a bit of restructure, just hoping to make it easy for ourselves to navigte

what do you mean an index that merges? they both benefit from access to the 10 const objects and fns; so those would have to be exported

[robertvanhoesel]

@robertvanhoesel are you comfortable with the file name change for technologies.ts to modifiers.ts? Would you be able to resolve the conflict when you are ready to pull?

If it's not too much, maybe best to create a folder 'modifiers' with index that merges abilities.ts and technologies.ts? I agree it needs a bit of restructure, just hoping to make it easy for ourselves to navigte

what do you mean an index that merges? they both benefit from access to the 10 const objects and fns; so those would have to be exported

In that case (you are right) probably best to have: /modifiers/index.ts if needed at all to merge and re-export all effects /modifiers/config.ts that contains the const and helpers /modifiers/abilities and /modifiers/technologies

do we want to delete classes property of abilities; currently all empty:

What do you think of setting them to ['Passive', 'Ability'] and ['Influence'] kinda classes?

[acstanton515]

@robertvanhoesel are you comfortable with the file name change for technologies.ts to modifiers.ts? Would you be able to resolve the conflict when you are ready to pull?

If it's not too much, maybe best to create a folder 'modifiers' with index that merges abilities.ts and technologies.ts? I agree it needs a bit of restructure, just hoping to make it easy for ourselves to navigte

what do you mean an index that merges? they both benefit from access to the 10 const objects and fns; so those would have to be exported

In that case (you are right) probably best to have: /modifiers/index.ts if needed at all to merge and re-export all effects /modifiers/config.ts that contains the const and helpers /modifiers/abilities and /modifiers/technologies

do we want to delete classes property of abilities; currently all empty:

What do you think of setting them to ['Passive', 'Ability'] and ['Influence'] kinda classes?

I'm struggling to understand reasoning for creating so many files for modifiers. I think the complexity is low with having it in a single file and the intent of the file is still clear. I am comfortable with the way it is. I'm happy to do this modification if you can show me what you would do in index.ts.

As to classing the abilities, I'd be happy to convert the "active" type into classes. Every ability has a class of ability and a class of the value we have in "active" today. Still more work to do in categorizing these abilities I think, in line with the other discussion we have on ToV.

Plus, some other topics to post.

[acstanton515]

There are maybe a dozen abilities that are unlocked by a tech (think: forced-march). Do we want to modify the tech effect to say "property":"unlockAbility", and then have the true effect in the abilities? Or do we exclude the "tech-ability" from abilities and keep all its data and ability data inside the tech?

There are some techs that upgrade abilities. How do we want to handle that? The tech could have "property":"upgradeAbility" (e.g. royal knight charge upgrade, network of citadels, khan whistling arrow)

[acstanton515]

There are some abilities that change with age. How do we want to handle that? (french vill production boost, english mill and farm rate, etc.)

[acstanton515]

I'm starting to look ahead at charge, spearwall, and secondary weapons. I'm thinking instead of charge and spearwall as an ability, to actually represent it as a property of unit. Logic will have to tie the secondary weapons to whether the unit can charge and/or has a spearwall. Just want to validate whether we should directionally add it to abilities or wait on investigating secondary weapons.

[acstanton515]

Don't know if you saw my comment about auto discovering abilities having some issues in the code:

//issue in these functions in that if the item was found with sbp then ebp wont discover, and vise versa
function findEbpAbilities(data: any) {
  const foundAbilities = data?.extensions?.find((x) => x.exts === "ebpextensions/ability_ext")?.abilities?.flatMap((u) => u.ability ?? u) ?? [];
  return foundAbilities;
}

function findSbpAbilities(data: any) {
  const foundAbilities = data?.extensions?.find((x) => x.exts === "sbpextensions/squad_ability_ext")?.abilities?.flatMap((u) => u.ability ?? u) ?? [];
  return foundAbilities;
}