search

akej74 / ksp-mod-analyzer

Analyze Kerbal Space Program mods available on SpaceDock and Curse
GNU General Public License v3.0
5 stars 1 forks source link

Web scraping from Curse is broken #7

Open akej74 opened 6 years ago

akej74 commented 6 years ago

Web scraping from Curse is currently broken due to changes on Curse page.

akej74 commented 6 years ago

Plan is to use the updated API at https://api.cfwidget.com to solve this.

HebaruSan commented 6 years ago

Hey @citricsquid, is there a way to use the widget to retrieve lists of mods rather than just info about specific mods?

shrink commented 6 years ago

@HebaruSan I have no plans to introduce an endpoint that would provide a complete list of what is available on CurseForge.

The API came about by accident (I originally planned to only provide widgets but figured why not? expose the data I had available for others to use — a mistake in hindsight) and I don't want to increase user dependency on my third-party API, although I do agree it would be very helpful to be able to programmatically retrieve a list of all mods.

I hope that someday the CurseForge team will see the demand and launch their own first-party API (including additional functionality like a list of all mods) so that I can shut my third-party API down. I would recommend that for any use case beyond "I have a mod identifier and need information about that mod" that developers scrape CurseForge themselves, and reach out to the CurseForge team to make their use case known.

I'm sorry that I can't provide a more helpful answer :-( I very much think this functionality should be available (and more!) but I have limited time and don't want to increase the surface area of what I'm responsible for, nor the amount of money I have to spend supporting the service. The API already makes tens of thousands of outbound requests per hour to CurseForge and supports tens of millions of inbound API requests per month. Any increase in that would require more resources committed on my part, in time and money, and I have other projects I'd like to use those resources on.

HebaruSan commented 6 years ago

Thanks! No need to further overburden your project; I just wanted to make sure we weren't missing an option that was already available. Cheers!

akej74 commented 6 years ago

Thanks for your input @HebaruSan and @citricsquid, I will look into doing this manually by web scraping the Curse page (using a threaded solution to be able to parse several mod pages in parallel).

akej74 commented 6 years ago

Hi @HebaruSan, @citricsquid, it looks like there is a new Curse API available, see the doc. Just like to check if you have used this...? I'm trying to figure out what can be done with the API; what I would need is to be able to retrieve data for all KSP mods (e.g. in JSON format or similar). Not sure if this is possible with the API...

HebaruSan commented 6 years ago

Hi @akej74, no, I haven't seen this before, thanks for letting us know. I don't see an API function that meets your needs at a glance, but good luck!

shrink commented 6 years ago

@akej74 That's the CurseForge mod author API, it has existed for many years. That API is for mod authors to manage their mods, it does not provide the functionality available via the cfwidget.com API. You can use that API to manage your own mods, but you cannot use it to retrieve public information.

akej74 commented 6 years ago

I see, thanks for the update @citricsquid.

shrink commented 6 years ago

I recently met with another user of the cfwidget API and we discussed expanding the API to include a full list of all projects available on CurseForge. Although I can't guarantee it will happen or when it will happen, it will probably happen soon. I'll make a note to update this issue when that functionality is available.

akej74 commented 6 years ago

Looking forward to that!

akej74 commented 5 years ago

Hi @citricsquid , just like to check if you have heard anyting on expanding the API...?

shrink commented 5 years ago

Hi @akej74,

Apologies, that previous comment doesn't reflect the current status of the API. Since my first comment here the API was temporarily shut down due to a block on scraping implemented by CurseForge, since then the API has been relaunched using a new data source which cannot be used to do anything beyond request projects on demand, meaning that any sort of indexing projects is not possible.

At this point, api.cfwidget.com should be considered maintenance only. I will ensure it remains online and functioning but I have no expectations that any additional features will be added. Ideally, at some point, an official curseforge.com API will be launched and api.cfwidget.com will be shut down after users have migrated.

I'm sorry that I don't have better news for you.

akej74 commented 5 years ago

Thanks @citricsquid for the info, let's see if there will be some official API in the future...