Missing counters for certain units

[gael-vanderlee]

Hi ! Love your tool, however, certain units (like the obuch) are missing counters and others (like the Charkram thrower) are missing entirely. I was wondering if it would be possible to use web scraping to extract all the units and their counters from the aoe2 wiki directly, to ensure it's always up to date. I could even write the web scraper in python if you think this'd be a good idea.

[lukethacoder]

Hi @Gvanderl

Glad you like the tool. Was just a quick weekend project initially. The data was pulled from some other aoe counter website, I can't quite remember, they just had the json file sitting there on an endpoint which made life easy. Building a scraper script to hit the official AOE2 Wiki sounds a lot more reliable, in terms of keeping this up to date. It would be awesome if we could click a button and get an updated list of units/counters.

Ideally this would be apart of the build/export step before publishing the site. But would be more than happy if you wanted to build a python based script, purely for scraping data and updating the units.json file.

If you think there is any extra data that the wiki provides that you think might be useful, I'd be open to expanding what data is stored/displayed on the site. (e.g. Age, Trained at etc.)

[gael-vanderlee]

Hi, I managed to get a nice little wiki scraper going. It's pretty simple for now but I'd like to get feedback before putting more time into it. Please let me know if there is any easily attainable info you'd like to have or if you'd like to change the format. I know the format isn't the same as the old one but this one makes more sense than an unordered list, and we don't have the data of the previous file so there are many fields I can't replicate. The counter units are plain strings, let me know if that's an issue but it's the cleanest form for now. I thought about creating a unique ID for each unit and list that ID instead of the unit name. This would complicate things and the result would be messy as unit counters on the wiki take many forms. There are unit names, armor types, technologies listed or just general description ("Weak vs everything" for example).

Here's a sample of the json copy pasted:

{
    "Teutonic Knight": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Teutonic_Knight",
        "strong_vs": [
            "Skirmishers",
            "melee units",
            "buildings"
        ],
        "weak_vs": [
            "Archers",
            "Scorpions",
            "Mangonels",
            "Hand Cannoneers",
            "Conquistadors",
            "Janissaries",
            "Slingers",
            "Monks",
            "Cataphracts",
            "Leiciai",
            "Obuchs",
            "Jaguar Warriors",
            "Samurai",
            "War Elephants",
            "Urumi Swordsmen (with Wootz Steel researched)"
        ]
    },
    "Janissary": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Janissary_(Age_of_Empires_II)",
        "strong_vs": [
            "Infantry",
            "buildings"
        ],
        "weak_vs": [
            "Skirmishers",
            "Scorpions",
            "Mangonels",
            "Huskarls",
            "Condottieri",
            "Bombard Cannons",
            "Winged Hussars"
        ]
    },
    "Berserk": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Berserk",
        "strong_vs": [
            "Skirmishers",
            "Halberdiers",
            "Camel Riders",
            "Eagle Warriors",
            "buildings",
            "Light Cavalry"
        ],
        "weak_vs": [
            "Cataphracts",
            "Jaguar Warriors",
            "Boyars",
            "Hand Cannoneers",
            "Conquistadors",
            "Slingers",
            "Plumed Archers",
            "Samurai",
            "Teutonic Knights",
            "mounted archers"
        ]
    },
    "Mangudai": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Mangudai_(Age_of_Empires_II)",
        "strong_vs": [
            "Infantry",
            "Villagers",
            "siege weapons"
        ],
        "weak_vs": [
            "Genoese Crossbowmen",
            "Huskarls",
            "Mamelukes",
            "Camel Riders",
            "Skirmishers",
            "Eagle Warriors",
            "Camel Archers",
            "Paladins",
            "Samurai",
            "Kamayuks"
        ]
    },
    "Longboat": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Longboat_(Age_of_Empires_II)",
        "strong_vs": [
            "Transport Ships",
            "Fishing Ships",
            "buildings",
            "land units in the shoreline and shallows",
            "Turtle Ships (hit-and-run only)",
            "Cannon Galleons",
            "Thirisadai (equal resources only)"
        ],
        "weak_vs": [
            "Fire Ships (only in small groups)",
            "Demolition Ships",
            "Samurai (in the shallows)",
            "towers",
            "Castles",
            "ranged siege units on the shoreline",
            "Caravels (in large fleet battles)"
        ]
    },
    "Eagle Warrior": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Eagle_Warrior",
        "strong_vs": [
            "Foot archers",
            "siege weapons",
            "Monks",
            "Genitours"
        ],
        "weak_vs": [
            "Gunpowder units",
            "Slingers",
            "Heavy infantry",
            "Heavy cavalry"
        ]
    },
    "Elite Eagle Warrior": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Elite_Eagle_Warrior",
        "strong_vs": [
            "Archers",
            "siege weapons",
            "Monks",
            "Halberdiers",
            "Heavy Camel Riders"
        ],
        "weak_vs": [
            "Heavy cavalry",
            "heavy infantry",
            "Gunpowder units",
            "Slingers"
        ]
    },
    "Halberdier": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Halberdier_(Age_of_Empires_II)",
        "strong_vs": [
            "Cavalry"
        ],
        "weak_vs": [
            "Archers",
            "Scorpions",
            "Hand Cannoneers",
            "Janissaries",
            "Monks",
            "infantry",
            "Slingers"
        ]
    },
    "Hussar": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Hussar_(Age_of_Empires_II)",
        "strong_vs": [
            "Archers",
            "siege weapons",
            "and Monks"
        ],
        "weak_vs": [
            "Most melee units",
            "Genoese Crossbowmen",
            "and Spearman-line"
        ]
    },
    "Petard": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Petard_(Age_of_Empires_II)",
        "strong_vs": [
            "Buildings",
            "siege weapons",
            "non-ranged units"
        ],
        "weak_vs": [
            "Ranged units"
        ]
    },
    "War Wagon": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/War_Wagon_(Age_of_Empires_II)",
        "strong_vs": [
            "Infantry",
            "archers",
            "buildings"
        ],
        "weak_vs": [
            "Cavalry",
            "Genoese Crossbowmen",
            "Huskarls",
            "Mamelukes",
            "Camel Riders",
            "Skirmishers",
            "Eagle Warriors",
            "Mangonels",
            "Monks"
        ]
    },
    "Turtle Ship": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Turtle_Ship",
        "strong_vs": [
            "Ships at close distance",
            "buildings on the shoreline",
            "units on the shoreline or shallows"
        ],
        "weak_vs": [
            "Bombard Cannons",
            "Monks",
            "Castles",
            "towers",
            "Condottieri (in the shallows)"
        ]
    },
    "Jaguar Warrior": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Jaguar_Warrior",
        "strong_vs": [
            "Skirmishers",
            "infantry",
            "buildings",
            "Camel Riders"
        ],
        "weak_vs": [
            "Archers",
            "Scorpions",
            "Mangonels",
            "heavy cavalry"
        ]
    },
    "Tarkan": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Tarkan",
        "strong_vs": [
            "Archers",
            "siege weapons",
            "Light Cavalry",
            "buildings"
        ],
        "weak_vs": [
            "Infantry",
            "heavy cavalry",
            "Genoese Crossbowmen",
            "Mamelukes",
            "Camel Riders"
        ]
    },
    "Missionary": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Missionary_(Age_of_Empires_II)",
        "strong_vs": [
            "Slow non-ranged units",
            "buildings without conversion defense",
            "Knights"
        ],
        "weak_vs": [
            "Archers",
            "Eagle Warriors",
            "Light Cavalry",
            "Camel Riders",
            "Halberdiers",
            "Genoese Crossbowmen",
            "Towers",
            "Castles"
        ]
    },
    "Conquistador": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Conquistador_(Age_of_Empires_II)",
        "strong_vs": [
            "Infantry",
            "Rams",
            "Buildings"
        ],
        "weak_vs": [
            "Genoese Crossbowmen",
            "Mamelukes",
            "Camel Riders",
            "Skirmishers",
            "Genitours",
            "Camel Archers"
        ]
    },
    "Plumed Archer": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Plumed_Archer",
        "strong_vs": [
            "Infantry",
            "archers",
            "Villagers",
            "Hand Cannoneers",
            "Camel Riders",
            "Mamelukes"
        ],
        "weak_vs": [
            "Skirmishers",
            "siege weapons",
            "cavalry",
            "Huskarls",
            "Rattan Archers",
            "Genitours",
            "War Wagons"
        ]
    },
    "Eagle Scout": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Eagle_Scout_(Age_of_Empires_II)",
        "strong_vs": [
            "Skirmishers",
            "Scout Cavalry",
            "Foot Archers"
        ],
        "weak_vs": [
            "Heavy Infantry",
            "Heavy Cavalry"
        ]
    },
    "Imperial Camel Rider": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Imperial_Camel_Rider",
        "strong_vs": [
            "Skirmishers",
            "siege weapons",
            "Cavalry Archers",
            "cavalry"
        ],
        "weak_vs": [
            "Pikemen",
            "Kamayuks",
            "Teutonic Knights",
            "Genoese Crossbowmen",
            "Berserks",
            "Cataphracts"
        ]
    },
    "Slinger": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Slinger_(Age_of_Empires_II)",
        "strong_vs": [
            "Infantry",
            "Villagers"
        ],
        "weak_vs": [
            "Skirmishers",
            "cavalry",
            "Genitours",
            "Rattan Archers",
            "Siege weapons"
        ]
    },
    "Genoese Crossbowman": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Genoese_Crossbowman",
        "strong_vs": [
            "Cavalry",
            "mounted archers",
            "infantry"
        ],
        "weak_vs": [
            "Foot archers (situationally)",
            "siege weapons",
            "Huskarls",
            "Eagle Warriors",
            "Ghulams",
            "Skirmishers"
        ]
    },
    "Condottiero": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Condottiero",
        "strong_vs": [
            "Skirmishers",
            "Hand Cannoneers",
            "Conquistadors",
            "Janissaries",
            "siege weapons",
            "Bombard Towers",
            "Bombard Cannons",
            "Light Cavalry",
            "Camel Riders",
            "Hussite Wagons"
        ],
        "weak_vs": [
            "Archers",
            "Scorpions",
            "Knights",
            "Boyars",
            "Teutonic Knights",
            "Samurai",
            "Cataphracts",
            "Jaguar Warriors",
            "Slingers",
            "War Elephants",
            "Battle Elephants"
        ]
    },
    "Elephant Archer": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Elephant_Archer_(Age_of_Empires_II)",
        "strong_vs": [
            "Slow non-ranged units",
            "archers",
            "buildings"
        ],
        "weak_vs": [
            "Spearmen",
            "Kamayuks",
            "Berserks",
            "Genoese Crossbowmen",
            "Huskarls",
            "Mamelukes",
            "Camel Riders",
            "Eagle Scouts",
            "Monks",
            "Skirmishers",
            "Scorpions",
            "Flaming Camels"
        ]
    },
    "Kamayuk": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Kamayuk",
        "strong_vs": [
            "Skirmishers",
            "cavalry (except Cataphracts)",
            "Spearmen",
            "Eagle Warriors"
        ],
        "weak_vs": [
            "Plumed Archers",
            "Scorpions",
            "Mangonels",
            "Hand Cannoneers",
            "Janissaries",
            "Slingers",
            "Cataphracts",
            "Teutonic Knights",
            "Jaguar Warriors",
            "Samurai",
            "mounted archers",
            "Mamelukes",
            "Rattan Archers"
        ]
    },
    "Magyar Huszar": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Magyar_Huszar",
        "strong_vs": [
            "Siege weapons",
            "archers",
            "Villagers"
        ],
        "weak_vs": [
            "Boyars",
            "Pikemen",
            "Kamayuks",
            "Berserks",
            "Genoese Crossbowmen",
            "Teutonic Knights",
            "Mamelukes",
            "Camel Riders",
            "Samurai"
        ]
    },
    "Boyar": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Boyar_(Age_of_Empires_II)",
        "strong_vs": [
            "Skirmishers",
            "melee units",
            "siege weapons"
        ],
        "weak_vs": [
            "Pikemen",
            "Kamayuks",
            "Teutonic Knights",
            "Mamelukes",
            "Camel Riders",
            "Monks",
            "War Elephants",
            "Genoese Crossbowmen",
            "Samurai",
            "Leiciai",
            "Obuchs"
        ]
    },
    "Siege Tower": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Siege_Tower_(Age_of_Empires_II)",
        "strong_vs": [
            "Nothing"
        ],
        "weak_vs": [
            "Melee units",
            "especially Magyar Huszars and Eagle Warriors"
        ]
    },
    "Fire Galley": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Fire_Galley_(Age_of_Empires_II)",
        "strong_vs": [
            "Transport Ships",
            "Fishing Ships",
            "buildings in the shoreline",
            "melee units in the shallows",
            "Galleys"
        ],
        "weak_vs": [
            "Demolition Rafts",
            "towers",
            "ranged siege units on the shoreline"
        ]
    },
    "Demolition Raft": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Demolition_Raft",
        "strong_vs": [
            "Transport Ships",
            "Fishing Ships",
            "Fire Galleys",
            "buildings in the shoreline",
            "melee units in the shallows"
        ],
        "weak_vs": [
            "Towers",
            "ranged units on the shoreline"
        ]
    },
    "Camel Archer": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Camel_Archer_(Age_of_Empires_II)",
        "strong_vs": [
            "Slow non-ranged units",
            "Cavalry Archers",
            "Conquistadors"
        ],
        "weak_vs": [
            "Huskarls",
            "Skirmishers",
            "Eagle Warriors",
            "Mangonels",
            "Camel Riders"
        ]
    },
    "Genitour": {
        "wiki_link": "https://ageofempires.fandom.com/wiki/Genitour",
        "strong_vs": [
            "Archers",
            "Cavalry Archers",
            "most Infantry"
        ],
        "weak_vs": [
            "Camel Riders",
            "siege units",
            "Huskarls",
            "Eagle Warriors",
            "Woad Raiders"
        ]
    }
}

[lukethacoder]

Hi @Gvanderl

Happy to change the format up, but it will require some changes to the site (not too big of an issue). An object of units does make more sense than an array of units. Key-wise, slugifying the names would probably be cleaner, (e.g. Teutonic Knight becomes teutonic-knight).

I do like that this dataset has both strong_vs and weak_vs. Only thing I can see we might be missing is the extra strings the dataset currently has - I assume someone has manually added these, not too much of an issue if these don't exist (e.g. "A mass of Mangudai can quickly take them down, but only with good micro to avoid their shots (which could be dangerous)"). Might be neat to somehow map up the above scraped data with the existing data. Slugifying the unit name might be a good way to tie them together

I assume things like the image url, category (infantry, calvary, naval etc.), sub category ("line", militia, spearmen, camel, knight, etc.) and civ (if unit is unique to) can be easily included as they are displayed on the wiki page we're scraping from. "Trained at", "Resource cost" and "Age" might be nice additions too.

I did some digging around and managed to find the original endpoint I stumbled across. It looks like it has since been updated. Our current dataset includes 63 units, whereas the live api link includes 100. Still no inclusion of the Charkram Thrower you mentioned initially.