Curse API is broken

[HebaruSan]

CKAN Version: HEAD

Operating System: Ubuntu 17.04

The issue you are experiencing:

http://status.ksp-ckan.org/

All mods hosted on Curse now say:

Could not get the mod from Curse, reason: project/ABCXYZ cannot be found on curse.com.

A test is also failing:

$ ./build test

1) Error : Tests.NetKAN.Sources.Curse.CurseApiTests.GetsModCorrectly
CKAN.Kraken : Could not get the mod from Curse, reason: Due to breaking changes at the curse.com website, this widget is currently available. Please visit the project page directly at: https://mods.curse.com/project/220221 Please email sryan@curse.com with any questions..
  at CKAN.NetKAN.Sources.Curse.CurseApi.GetMod (System.Int32 modId) [0x00034] in <896b7ef4c8e0431fb2e00da75ce0c276>:0 
  at Tests.NetKAN.Sources.Curse.CurseApiTests.GetsModCorrectly () [0x00012] in <fc05fcb18e70447b9a563b5729c30b33>:0 
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in <8f2c484307284b51944a1a13a14c0266>:0 

Extracting the error for readability:

Due to breaking changes at the curse.com website, this widget is currently available. Please visit the project page directly at: https://mods.curse.com/project/220221 Please email sryan@curse.com with any questions..

The client can still download and install mods hosted on Curse. It's just the checking for new updates that's broken.

Interesting timing just days after GitHub broke travis...

[HebaruSan]

https://www.curseforge.com/docs/api is now a 404. Did they just sunset their API completely?

Old copy at archive.org for reference: https://web.archive.org/web/20170919230707/https://www.curseforge.com/docs/api

[HebaruSan]

Looks like there are only six 1.3.1-compatible mods that have a $kref containing curse (and therefore will no longer auto-update):

• ForScience
• HumanColoredFaces
• OPTSpacePlaneMain
• RealisticAtmospheres
• StoreMyReports
• SamBelangerFlags

[Olympic1]

That is because we depend on a widget that pulls information from curse. Due to changes in the Curse API, it is unavailable.

[HebaruSan]

So that's a third party tool that parses Curse's HTML? I guess that means they might fix it.

[HebaruSan]

Guess where the widget's source is hosted? :wink:

https://github.com/citricsquid/widget/issues/44

Apparently the site's structure changed to be much more time-intensive to crawl and may require the widget to be significantly more complicated. But they do seem to be planning to fix it.

[shrink]

Hi! I'm the maintainer of the widget :-) I don't have an accurate ETA but I am working on fixing the widget, and it should be back soon — this week, hopefully. Previously the widget was very simple, as all the data was on one page: request comes in -> scrape the mods.curse.com project page -> return the result... however, the data is now split across many pages and so I will need to switch to background workers to process project data, which adds complexity.

The aim is to ensure backwards compatibility, so your implementation should not need to change, however, the new CurseForge website does not include all of the information previously available therefore there will be some changes. Specifically, I believe that you make use of the file license information which the widget api previously provided, however, this is no longer available through the CurseForge website and therefore I will no longer be able to return an accurate value for this field.

My intention with the new API is to return all fields that were previously included, but where data is no longer available I will replace the value with something like "n/a". If there's a specific value that you'd like me to return for the license field that will allow your implementation to continue to work, please let me know and I will make sure to include that.

Let me know if you have any other concerns.

Also for the sake of clarity, for anyone unfamiliar: I am a Curse employee, however, I run the widget as an independent third party and was not given notice that the curse website was changing which meant the breaking changes came as a surprise to me. I am quickly working to resolve the issues as the widget api handles tens of millions of requests per month and the widgets themselves are delivered many millions of times. I am very sorry for the inconvenience caused by the api breaking, I am hopeful that eventually, CurseForge will provide a first-party API that means you will no longer need to depend on my project.

[HebaruSan]

Thanks for the updates, @citricsquid!

No need to apologize or hurry; a very small proportion of our mod index is affected, the impact on them is minor, it would have been much more difficult to handle this all by ourselves, and (obviously) it's not your fault anyway.

On the subject of licenses, I think it would be about the same for us if you chose "n/a" or "" or null. We require mods to have known licenses to be indexed, but if we can no longer get them from the widget, we already have other ways of handling overrides.

Thanks again for being responsive!

[shrink]

https://api.cfwidget.com/kerbal/ksp-mods/kerbal-attachment-system-kas

The widget and API are back online, it now lives at https://api.cfwidget.com -- replace curseforge.com with api.cfwidget.com in any URLs. The output isn't exactly the same so you may need to make some (minor) tweaks but it should roughly include the same information.

Let me know if you run into any issues.

Thanks for the patience!

[HebaruSan]

Hey @citricsquid , thanks for the update!

Our database currently stores ID numbers for Curse mods, but it looks like the new URLs use names instead of numbers. Is there any way we can get to the new data via the ID numbers? If not, we can update our database to use the names instead, but if there's a way to avoid that, that would probably be the better option.

[shrink]

@HebaruSan the widget supports any valid CurseForge URL, which includes project/{id}, e.g: https://api.cfwidget.com/project/223900. Try that :) Should be working now, just deployed a change.