search

Danacus / node-factorio-api

Download and update mods from the Factorio Mod Portal
3 stars 2 forks source link

New mod portal changes broke things #4

Closed Danielv123 closed 6 years ago

Danielv123 commented 6 years ago

The mod portal has just recieved a massive overhaul, API included as they have gone away from basing the core of the site on AJAX calls.

Now, these api changes have also broken most tools relying on them, including this project 😢

Among the things I have noticed are broken:

Probably more, but those are the ones I have caught this far. Also, you still haven't pushed to npm after resolving #2 😄

Danacus commented 6 years ago

Oh sorry, I totally forgot about that.

I might try to update this whenever I've got time. It will be tricky, because it's not documented for as far as I know. I'll have to rewrite a lot of things probably. I do want to maintain this project.

And may I ask where you're using this for? I'm just curious :)

Danielv123 commented 6 years ago

Found some info regarding the searching issue!

< mickael9> looks like you can't use /api/mods?q=SEARCH to search mods now :(

I am using this for managing mods from the commandline, also downloading mods before joining a server so I don't have to mess with multiple installs. I really don't like using the browser for file management...

Danacus commented 6 years ago

Oh yeah I get it, makes sense to use it that way. Managing mods in-game isn't really the greatest option either.

I'm basically just going to try to figure out which requests are being made while I'm using the mod portal website. That's how I figured it out in the past. I might also improve my unit testing and make sure everything works again.

Danielv123 commented 6 years ago

I believe /u/bilka2 has posted some information about the portal API on github, he might have more info in case you want to ask him.

Danacus commented 6 years ago

I'm not sure where to find that information, but I might ask him if I need help.

Danielv123 commented 6 years ago

Found some more info from people talking about the changes maybe.

https://pastebin.com/hS8wJHbj

Danacus commented 6 years ago

Yeah, it might be better to wait until they (finally) make a public API. That would make things easier anyway. The previous API wasn't public, but the community documented it on the wiki. I don't want to waste my time figuring it out right now, as it will probably change anyway.

Danacus commented 6 years ago

Hey Daniel

I've got some good news and some bad news. The good news is:

node_factorio_api

The bad news is, searching mods is no longer possible. Apparently, the Factorio devs never used this functionality, because Factorio just gets a list of all the mods and searches in that list locally. That's probably why they removed it. Anyway, I might do something similar, so you can search mods again.

Fixing the problem with the file names was very easy. To fix the dependency problem, it now uses the local zip file from a mod to get it's dependencies. Reading mods from a save file, should work with 0.16 saves, but not with older versions.

The update is already published to npm and I'll try to find a solution for searching mods.

Danacus commented 6 years ago

Okay, you can ignore the bad news. I fixed the searching problem by just requesting for the first 100000 mods and filtering them manually. I sorted them from most to least downloads. The parameter of the searchMods function is now just a query and not an object.

Danielv123 commented 6 years ago

Great work, this made my day so much better!

Just wondering, is the speed of searches significantly impaired by having to fetch all of the mods? How many mods are there even?

Danacus commented 6 years ago

It used to be very slow and I expected it to still be very slow, but it has improved a lot. You may have noticed that when you try to download mods in-game, it loads a lot faster now too. I don't know how many mods there are, but I'm pretty sure it's less than 100000. I might have to change that number in a couple of ages xD. You can try this link if you want to see how fast it is: https://mods.factorio.com/api/mods?page_size=100000

Danielv123 commented 6 years ago

According to my dev console, there are 2535 mods, which results in 1.1 mb of data which downloads in 3 seconds. For a full 100k mods, that would be about 40mb of data, which is still not too bad for something that gets downloaded so rarely. And this is probably a thing we could cache pretty safely.

Danacus commented 6 years ago

Yeah, I could make it so that it only downloads once when you authenticate and caches it so it doesn't have to download it again for each search query. But 1.1 MB isn't that bad I suppose.

Danielv123 commented 6 years ago

Personally, I think that caching should be done by the user. You might want to for example recheck to see if a mod has been updated etc, and having to deauthenticate for that would be confusing.

Danacus commented 6 years ago

Yeah, I agree. I could make a function to reload cached data. The CheckUpdates function is not affected by the changes of the mod portal, because you can still find information about a specific mod if you have the exact name. Every time you call CheckUpdate it will go to a link like this one: https://mods.factorio.com/api/mods/boblogistics . It doesn't load all mods to check updates. The fix we're talking about was only for search queries. I think the chance a new mod gets posted right after you authenticate is very small. If download a mod, it will always try to download the latest mod, it doesn't matter if you just reloaded the list of all the mods. It would still make sense to have a function to update the cached data anyway. I hope any of what I said makes sense.