Updates and open issues

[slemke]

Hey @seirl and @kevinclement,

Are you looking for some help in maintaining this project? I am looking for an OS project to sink some time into. I would also like to use this project in shadowlands for myself and with the open issues and stuff I think I could do some good here.

[seirl]

Thank you a lot for this proposal!

I think so, yes. I really need to focus on other IRL stuff right now so I won't play shadowlands. I sent an email to kevin to tell him that I would be a bit busy for the time being, and I don't know if he has time to work on it either. I haven't had time to update all the python import scripts I wrote to use the new (horrible) API, so there's like... technical debt on top of the usual hard work required to do the data import.

I'd be very happy to review and merge data entry pull requests! It would be a shame if people couldn't use SimpleArmory for Shadowlands...

[slemke]

Great! I will have a look at the import scripts over the next few days, but I probably can't do much for the pre patch this week.

Regarding angular, bower and the other frontend stuff, do you have any plans on updating / migrating to a newer version? I could get rid of the old bower dependency for now, but anything beyond that would probably be a very large change.

[seirl]

I have absolutely no idea, I know next to nothing in frontend / the js ecosystem, I'll let Kevin reply when he sees this

[slemke]

I had a look at the mount script and tried to update the current mount data. So ... yeah you weren't kidding about the new API. It seems that blizzard removed some data from the mounts endpoint which now only returns entries like this:

    {
      "key": {
        "href": "https://eu.api.blizzard.com/data/wow/mount/6?namespace=static-9.0.1_36072-eu"
      },
      "name": "Brown Horse",
      "id": 6
    },

Stuff like spellid and icon is gone. It looks like I can get this back if I use addditonal csv files from wowtools. I would probably change the current setup to something like this:

1. get the mount list from the battle.net API
2. use the APIs mountID to look up SourceSpellID here: https://wow.tools/dbc/?dbc=mount&build=9.0.2.36267
3. use the SourceSpellID to get the itemID here: https://wow.tools/dbc/?dbc=itemtomountspell&build=6.0.1.18179
4. use the itemID to get the IconFileDataID here: https://wow.tools/dbc/?dbc=item&build=9.0.2.36267
5. use the IconFileDataID to get the filename: https://wow.tools/dbc/?dbc=filedata
6. remove the file type (.blp) and make the filename lowercase
7. aggregate the data into the mount list
8. have a list with unobtainable mounts and mark them in the mount list
9. have a list with mounts to ignore (from mounts_fixer_bnet.py)
10. transform the list of mounts into a structure for the frontend

With this we could generate a new mount.json repeatedly in a decent semi-automated way. I would rather get this data from the battle.net API itself, but even the endpoint for a specific mount (using the id) does not return this stuff. What do you think? With the current design of the battle.net API I don't see another option to make this work.

[Marcarrian]

Hey @seirl and @slemke A while ago I wrote a mail to Kevin asking if he has any plans on updating the project to Angular 10, to which he didn't reply. I then started anyway, pushing the changes to a feature branch on my forked repository. I'm nowhere near done. I have the basic project structure ready and implemented certain components (mounts, companions, battlepets). While simultaneously removing old stuff (like the bower dependency you mentioned) and much more.

If this is similar to what you had in mind @slemke (or anyone else who likes to contribute), then I would love to have a talk with you and discuss the next steps.

At the same time I see the risk of such a huge change. Hence in my opinion the number one priority should be to prepare SimpleArmory for the Shadowlands release.

[slemke]

Unless there is a comment from @kevinclement or @seirl, there won't be any next steps (atleast for me). They obviously had some ideas when building this and I am happy to help out with those ideas and maintenance stuff, because I like to use SimpleArmory myself. But the thing with OS projects is, if there is not some sort of discussion about bigger changes, it will mostly be wasted time. This is why I commented my approach above.

Yes, having the data for shadowlands should be the first priority. Anything else can be done later (or not at all - if not forced otherwise by a third party, like bower not working anymore). But this all means nothing if there is no one to review this with atleast a collaborator status. For now I try to fix the data. Beyond that I think @kevinclement and @seirl need to have a discussion how they want to handle the future of SimpleArmory or at least find out how to handle PRs.

Edit: From what I can tell the angular update won't work anyway, because the build requires grunt and there is no config for the cloud build in the repository. So the changes shouldn't take effect unless someone with access changes the cloud build script.

Edit 2: To maybe make this easier: I am willing to help out as a collaborator. If you are interested I can provide an email/discord/battle.net id so we can talk about it.

[seirl]

To be clear I'm mostly doing data entry and maintenance work, I didn't have any significant part in building the frontend stuff, and I don't want people to assume I'm in charge just because I have committer rights. :stuck_out_tongue:

I agree that the current priority is data entry for Shadowlands, and the rest of the discussion can happen later down the road.

[Marcarrian]

I didn't know about wow.tools so apologize if I'm making any obvious mistakes. I tested your suggested procedure of aggregating the mount data and found some rather worrying results:

Example Lucid Nightmare: ID 961 -> SourceSpellID 247402 link SourceSpellID 247402 -> no ItemID found for provided SpellID link

Example White Polar Bear: ID 237-> SourceSpellID 54753 link SourceSpellID 54753 -> ItemID 40777 and 43962 link ItemID 40777 -> IconFileDataID 132117 link ItemID 43962 -> IconFileDataID 236245 link IconFileDataID 132117 -> Filename Ability_Druid_ChallangingRoar.blp link (obviously wrong) IconFileDataID 236245 -> Filename Ability_Mount_PolarBear_White.blp link (correct)

[kevinclement]

Sorry for delay folks, I am around, just life of course gets busy it seems. I'm planning to play in shadowlands, but didn't have time to sort it out before pre-patch.

As folks are noticing, getting this info from blizzard can be pretty tiresome and at times requires hacks to try and piece it all together. That last api change was pretty bad in they broke some fundamental link points in the API, kind of sad.

I've had this idea in the past to pivot a bit and maybe it's time to dig into it more. I wrote some proof of concept code that did a database dump from within the game using an addon and then could save that out into a file. From that file you could import into our "db" files. At the time it was to work around "whats in the game versus isn't" but might be a better work around for the API craziness.

I can't remember now since it's been a while since I dusted off this part of my brain. I think I used or looked at wow.tools and that might be a better way too. Haven't had time to sit down with it just yet.

[kevinclement]

Great! I will have a look at the import scripts over the next few days, but I probably can't do much for the pre patch this week.

Regarding angular, bower and the other frontend stuff, do you have any plans on updating / migrating to a newer version? I could get rid of the old bower dependency for now, but anything beyond that would probably be a very large change.

I had a branch for this at one point and it ended up being a bit of a rats nest that I didn't have time to sort out. End result was I left it alone. I'd like to get rid of the older tech like bower that is clearly past it's age. However, I'm not as sold on updating angular. There is reason to update, and I'd like to, but it's not clear that I'd keep with angular and I'd want to explorer other options. going from this version of angular to newer could be as big an undertaking as just porting it.

[slemke]

I didn't know about wow.tools so apologize if I'm making any obvious mistakes. I tested your suggested procedure of aggregating the mount data and found some rather worrying results:

Yeah, I just noticed the itemtomountspell data is from 6.0.1, which obviously won't work.

However, I'm not as sold on updating angular. There is reason to update, and I'd like to, but it's not clear that I'd keep with angular and I'd want to explorer other options. going from this version of angular to newer could be as big an undertaking as just porting it.

If you have something else in mind and you want a helping hand I have experience with most of the "current popular frameworks" (angular, react and vue). I suggested an angular update mostly because the project currently uses it.

I've had this idea in the past to pivot a bit and maybe it's time to dig into it more. I wrote some proof of concept code that did a database dump from within the game using an addon and then could save that out into a file. From that file you could import into our "db" files.

I also tried a proof of concept like this yesterday, but with the battle.net API instead. Based on the current mounts.json I generated a mountlist and added the ids from the the battle.net API, so I can check for new mounts. This is kinda like the dump you mentioned. I then used BeautifulSoup to get the other data (itemId, icon, ...) from wowhead, which seems to work. I need to verify this further, but this could work too. If you want I can do a PR by the weekend so you can have a look.

[seirl]

@slemke I suggest using wowdb instead of scraping wowhead. Check out the battle pets script I wrote, it uses wowdb to fetch the pet data.

Ideally it would be good if we could just the scripts I wrote to use the new API and wow.tools without changing the way the scripts are used. We really don't want to do manual updates at each patch.

@kevinclement I updated the addon, so doing this is still possible using /simplearmory dev {mounts,pets,toys}. However after doing some testing it misses all the "hidden" items so it's not really useful to check if you're missing anything. The API doesn't have this problem.

[seirl]

I suggest we use this topic for the Shadowlands data import, and maybe we can create another thread for the frontend tech update stuff?

[seirl]

• use the SourceSpellID to get the itemID here: https://wow.tools/dbc/?dbc=itemtomountspell&build=6.0.1.18179

Ugh... I don't think that step even works. As shown in the URL, this file hasn't been updated since WoD.

[seirl]

The achievements are done! b3ed5a2

[slemke]

I added the mount data in #299.