classic versions of addons being installed

[torkus]

For example, DBM is installing "1.13.0 - classic" instead of "8.2.12" (DBM page here: https://www.curseforge.com/wow/addons/deadly-boss-mods/files)

And BigWigs boss mods is doing something similar.

From the scraper's POV it's the most recent release on Curseforge rather than a different release.

At time of writing you can install slightly older versions of DBM and Bigwigs from wowinterface.

I'm sure there are others.

~I have no fix or idea on how to accommodate it yet. Just like Curseforge, apparently.~

update 2019-08-30

I have a development version of wowman with support for switching between 'classic' and 'retail' modes of WoW. It needs some polish and unit tests which will happen at my own pace, but if you'd like to give it a go you can download it here:

wowman-0.9.0-unreleased-standalone.jar.zip

sha256: 11d380cef0cdf31ded29a95688ec8983c69f3d971afb6819fd3db57b70bebb71

(you'll need to remove the '.zip' suffix. Github doesn't let you upload .jar files)

[layday]

I wonder how the Twitch app handles classic - does their API discriminate between retail and classic?

[torkus]

I've never used it, I'm not sure how it behaves. What I'm thinking is:

• scrape the list of versions available
• curseforge has a release 'type' (like 'beta') that could become one of several simple 'tags' for a release.
• look for other tags in the release, like 'classic'
• allow the user to beta/alpha/classic releases
• allow the user to opt out of beta/alpha/classic releases

This would handle those not playing Classic.

And if the user can opt-out of certain tagged releases, then they can do the inverse and opt-in, only ever receiving the latest beta/alpha/classic version of an addon. This is where it breaks down a bit though, because a 'classic' release of an addon is really a separate addon munged into the regular stream of releases.

https://www.curseforge.com/wow/addons/deadly-boss-mods/files?sort=-releasetype

I'm curious to know what Curseforge have planned, if anything. It would make sense to create a different 'release type' for classic versions of an addon.

[torkus]

Although ... looking at that screenshot, I could differentiate based on 'game version'

[torkus]

I'm not sure if WoWInterface are enforcing it somehow, but DBM has a separate addon there:

[layday]

If you're not dead set on scraping, the Twitch files API does contain a gameVersionFlavor field which can be either of: wow_retail, wow_classic. See [1] for DBM. The response body can be quite large. I expect that their addon API will be updated to contain classic versions of add-ons sometime in the not-too-distant future so we don't have to sift through the entire file list.

[torkus]

I don't know much about these APIs - are they official sources?

[layday]

That's the API that the Twitch client uses internally. It has been documented by a third party here.

[torkus]

cheers, I came across this before while going through your code.

I'll take a closer look and see if it can be adapted to how wowman works.

[layday]

P.S. I like what you've done with wowman and I think it's great that you've catalogued related work.

[torkus]

thank you :) I really appreciate the compliment.

I spent about a year building wowman before I squashed the changes and made it public in April. About that time, on a whim, I started looking for 'competition' and found heaps. I was a little unhappy and discouraged but realised so many had fallen by the wayside. There was often a lot of enthusiasm to begin with, lots of promises and 'coming in 2.0' etc, but much fewer were still being regularly updated. I built yet-another-addon-manager to see a better class of software for Linux gamers and as a hobby, which is probably why most of these other projects exists, and I bet most were all just as oblivious to each other as I was.

Hopefully a quick search turns up that list for the next hopeful and they decide to collaborate instead of building their own from scratch.

[torkus]

I should do more with that list, actually. Some of them have strong features that really distinguish them and some of them are pretty weak but are getting equal billing. I recommend your own instawow as a go-to for those who prefer the CLI.

wowman isn't Python but I work with it a lot. I wish it were more self contained.

[torkus]

I can't get that api search function to work for me. I have what I think are the correct values but nothing comes out. I see a bit of a discussion here - do you have a working url?

[layday]

https://addons-ecs.forgesvc.net/api/v2/addon/search?gameId=1&index=0&pageSize=255&searchFilter=handynotes+mechagon&sort=2 from that thread still works for me.

[torkus]

cheers, I'll take a closer look

[torkus]

it's crazy, I just requested 5000 addons, a 30MB download. I think index is an offset into the results but I get 404s on high numbers, like 5000. I might just keep it to 255 and page that way. So much lovely data ...

[layday]

I expect that their addon API will be updated to contain classic versions of add-ons sometime in the not-too-distant future so we don't have to sift through the entire file list.

Looks like they've done just that earlier today (breaking instawow in the process :).

[torkus]

do you mean this endpoint: https://addons-ecs.forgesvc.net/api/v2/addon/{project-id} ?

I have output from it from last night and the same just now and the response bodies are identical

[layday]

I do - if you attempt to retrieve an add-on that supports Classic (e.g. BigWigs) both retail and classic versions will be included in latestFiles and gameVersionLatestFiles. That wasn't the case yesterday (I think).

[torkus]

ah, nice catch then. was latestFiles previously retail-only?

[torkus]

I'm trying to resolve a discrepancy of about 235 addons that appear on the website but are not present in the API results. Apart from that the catalog generation is going well.

[layday]

Do you have an example? It could be they exclude unmaintained add-ons and/or add-ons without files from API results.

ah, nice catch then. was latestFiles previously retail-only?

Yes, indeed.

[torkus]

it's none of those unfortunately, nothing common between them. They don't show up in search results if you search for them explicitly either. The API can access them directly via the /addon endpoint.

I've been comparing these against addons that are being returned by the api:

  }, {
    "alt-name" : "addonchecker",
    "category-list" : [ "Miscellaneous" ],
    "created-date" : "2010-05-27T22:45:55Z",
    "description" : "Addon Checker checks for common raiding addon to ensure they are upto date.",
    "download-count" : 355,
    "label" : "Addon Checker",
    "name" : "addon-checker",
    "updated-date" : "2010-05-27T23:31:15Z",
    "uri" : "https://www.curseforge.com/wow/addons/addon-checker"
  }, {
  }, {
    "alt-name" : "addonskinsclassic",
    "category-list" : [ "Plugins" ],
    "created-date" : "2019-06-23T08:54:45Z",
    "description" : "Collection of Skins for User Interfaces at Tukui.org",
    "download-count" : 402,
    "label" : "AddOnSkins Classic",
    "name" : "addonskins-classic",
    "updated-date" : "2019-08-10T04:08:54Z",
    "uri" : "https://www.curseforge.com/wow/addons/addonskins-classic"
  }, {
  }, {
    "alt-name" : "ahexport",
    "category-list" : [ "Auction & Economy", "Data Export", "Professions" ],
    "created-date" : "2010-04-09T08:49:07Z",
    "description" : "Easily export the entire Auction House data",
    "download-count" : 366,
    "label" : "AHExport",
    "name" : "ahexport",
    "updated-date" : "2010-04-09T08:49:07Z",
    "uri" : "https://www.curseforge.com/wow/addons/ahexport"
  }, {
  }, {
    "alt-name" : "ak47hunteraudiopack",
    "category-list" : [ "Audio & Video", "Combat", "Hunter", "Damage Dealer", "Achievements" ],
    "created-date" : "2015-06-07T14:43:39Z",
    "description" : "Make your guns roar like an Ak47",
    "download-count" : 519,
    "label" : "AK47 Hunter Audio Pack",
    "name" : "ak47-hunter-audio-pack",
    "updated-date" : "2015-06-07T14:44:15Z",
    "uri" : "https://www.curseforge.com/wow/addons/ak47-hunter-audio-pack"
  }, {
  }, {
    "alt-name" : "alachatclassic",
    "category-list" : [ "Chat & Communication" ],
    "created-date" : "2019-06-21T16:07:06Z",
    "description" : "A lightweight chat ulitiy for wow classic. LOW cpu & mem cost.",
    "download-count" : 374,
    "label" : "alaChat_Classic",
    "name" : "alachat_classic",
    "updated-date" : "2019-08-10T05:21:22Z",
    "uri" : "https://www.curseforge.com/wow/addons/alachat_classic"
  }, {
  }, {
    "alt-name" : "allyroute",
    "category-list" : [ "Quests & Leveling" ],
    "created-date" : "2019-07-11T03:23:24Z",
    "description" : "In-game Alliance Leveling Guide. 1-60 Step by Step.",
    "download-count" : 1300,
    "label" : "AllyRoute",
    "name" : "allyroute",
    "updated-date" : "2019-07-11T11:13:39Z",
    "uri" : "https://www.curseforge.com/wow/addons/allyroute"
  }, {
    }, {
    "alt-name" : "annoyingsounds",
    "category-list" : [ "Audio & Video" ],
    "created-date" : "2011-02-08T14:45:43Z",
    "description" : "AnnoyingSounds",
    "download-count" : 859,
    "label" : "AnnoyingSounds",
    "name" : "annoyingsounds",
    "updated-date" : "2015-04-21T02:31:21Z",
    "uri" : "https://www.curseforge.com/wow/addons/annoyingsounds"
  }, {

    }, {
    "alt-name" : "atlasworldmap",
    "category-list" : [ "Map & Minimap" ],
    "created-date" : "2014-12-14T12:40:21Z",
    "description" : "World Map with Minimap style",
    "download-count" : 33200,
    "label" : "Atlas World Map",
    "name" : "atlas-world-map",
    "updated-date" : "2016-08-29T23:07:00Z",
    "uri" : "https://www.curseforge.com/wow/addons/atlas-world-map"
  }, {
  }, {
    "alt-name" : "atlaslootclassic",
    "category-list" : [ "Boss Encounters", "Map & Minimap", "Professions", "Arena" ],
    "created-date" : "2019-07-07T09:34:57Z",
    "description" : "Shows the possible loot from the bosses",
    "download-count" : 11800,
    "label" : "AtlasLootClassic",
    "name" : "atlaslootclassic",
    "updated-date" : "2019-08-10T12:14:25Z",
    "uri" : "https://www.curseforge.com/wow/addons/atlaslootclassic"
  }, {
  }, {
    "alt-name" : "autoconfirmmerchanttrade",
    "category-list" : [ "Bags & Inventory" ],
    "created-date" : "2016-08-05T19:33:58Z",
    "description" : "Skip Pesky Popup",
    "download-count" : 434,
    "label" : "AutoConfirmMerchantTrade",
    "name" : "autoconfirmmerchanttrade",
    "updated-date" : "2016-08-05T21:43:16Z",
    "uri" : "https://www.curseforge.com/wow/addons/autoconfirmmerchanttrade"
  }, {

  }, {
    "alt-name" : "buffedtankstats",
    "category-list" : [ "Tank" ],
    "created-date" : "2011-05-05T22:21:37Z",
    "description" : "Buffed Tank Stats",
    "download-count" : 144,
    "label" : "Buffed Tank Stats",
    "name" : "buffed-tank-stats",
    "updated-date" : "2011-05-05T22:21:37Z",
    "uri" : "https://www.curseforge.com/wow/addons/buffed-tank-stats"
  }, {
    "alt-name" : "buffenoughclassic",
    "category-list" : [ "Buffs & Debuffs" ],
    "created-date" : "2019-06-21T06:24:24Z",
    "description" : "Checks your buffs are ready!",
    "download-count" : 412,
    "label" : "BuffEnoughClassic",
    "name" : "buffenoughclassic",
    "updated-date" : "2019-08-09T06:58:20Z",
    "uri" : "https://www.curseforge.com/wow/addons/buffenoughclassic"
  }, {  

and a script to dump and order the responses so I can diff them:

import json
from pprint import pprint
import requests

rows = [
    ("anewsimplepercent", 319346), # control

    ("addonchecker", 23788),
    ("addonskinsclassic", 325109),
    ("ahexport", 22776),
    ("ak47hunteraudiopack", 92489),
    ("alachatclassic", 324936),
    ("allyroute", 327019),
    ("annoyingsounds", 28946),
    ("atlasworldmap", 87764),
    ("atlaslootclassic", 326516)
]

def api_url(path, *args):
    path = "https://addons-ecs.forgesvc.net/api/v2" + path
    return path % args

def fetch(name, pid):
    print('fetching',name)
    results = requests.get(api_url("/addon/%s", pid))
    data = results.json()
    data = sorted(data.items())
    json.dump(data, open(name + ".json", 'w'), indent=4)
    print('wrote', name + '.json')
    pprint(data)

for nom, pid in rows:
    fetch(nom, pid)

[torkus]

As lovely and convenient as the API is, I can't have it mysteriously excluding random addons. I'll add the project ID to the html scraping and then use the api when fetching detailed addon info.

[layday]

Did you have a list (or extract) of add-ons that are excluded by any chance? There was interest for adding a search function to instawow so I would like to know more about this.

[antiwinter]

Hello, sorry to interrupt :)

I am currently working on the command line version: wowa. When I make requests to wowinterface.com, the request is blocked by cloudflare sometimes. But I have tried your wowman and it can always make requests successfully when you scraping the catalog. Do you have any special settings on this?

[torkus]

hi, this issue has turned into a general conversation so I don't mind the interruption ;)

wowinterface.com is very quick and easy to scrape and I've never had a problem. I last did a complete scrape yesterday evening without problem (I haven't pushed the update yet because I'm debugging this api issue).

I've noticed curseforge.com give my current web browser some checks in the last few days.

[torkus]

hrm, now that you mention it, I can see something fishy is going on talking to curseforge

[antiwinter]

Yes, curseforge does have a browser check for the search function several weeks ago. Maybe they are going to add this feature to other pages. But I think this check can be bypassed by using a scraper that can execute javascript.

For wowinterface it is even worse, the check mechanism provides a picture and you need to input the captcha...

[torkus]

huh - unhandled scraping case: https://www.curseforge.com/wow/addons/muffinfactionizer/files

huh - it looks pretty borked actually. 404 on the download link as well. I wonder what they did. Muffinfactionizer has been on my test list for a long time

[torkus]

Yes, curseforge does have a browser check for the search function several weeks ago.

I think that is what I was seeing.

Attempting to download the first 39 wowinterface addons through wowman ... done. No problems (except for some dodgy addons wowman now refuses to install).

[torkus]

Pretty sure wowinterface have an API you can use, @antiwinter . @layday is using it. If you're having problems with the HTML you might want to switch sources

[torkus]

Github Issues is not the best place to chat.

I'm on #wow-addon-manager on freenode.net if anybody wants to chat there.

Online IRC client here: https://webchat.freenode.net/

Click the + once you pick a nick and enter "wow-addon-manager"

[torkus]

work on this fix/feature is happening here: https://github.com/ogri-la/wowman/pull/31

I won't be doing anything too clever to support classic addons. I've introduced support for multiple addon directories and each addon directory can be either 'retail' or 'classic'.

If 'classic' is selected (default is 'retail'), addons whose interface-version is incompatible will be highlighted. The user can then opt to uninstall (or not) that addon.

The search function is being modified to filter search results against the currently selected retail/classic mode. There is enough addon information between wowinterface and curseforge to accommodate this consistently.

[torkus]

to any who are interested: I've updated the description of this ticket with a development version of wowman that supports switching between classic and retail modes.

[layday]

Since this has turned into a bit of a catch-all, I thought I'd mention there's an option under 'Watch' to be notified of releases only.

[torkus]

0.9.0 has just been released that includes support for installing classic addons

thanks, all