search

KSP-CKAN / NetKAN

Metadata files used by the NetKAN/CKAN indexer for KSP
Creative Commons Zero v1.0 Universal
72 stars 340 forks source link

[Bug] 404 errors for Curseforge hosted mods #7268

Closed DasSkelett closed 5 years ago

DasSkelett commented 5 years ago

Problem

Describe the bug See status.ksp-ckan.space, (all?) mods hosted on Curseforge are failing with The remote server returned an error: (404) Not Found.. Get the same error inflating locally.

The JSON returned by the API URL (e.g. https://api.cfwidget.com/project/283006) looks fine though.

NetKAN stacktrace, useful for example API URL (mod DiscordRP):

Click to expand

516 [1] INFO CKAN.NetKAN.Sources.Curse.CurseApi (null) - Calling https://api.cfwidget.com/project/283006
516 [1] DEBUG CKAN.Net (null) - About to download https://api.cfwidget.com/project/283006
2424 [1] DEBUG CKAN.NetKAN.Processors.Inflator (null) - Deleting downloads for failed inflation
2425 [1] FATAL CKAN.NetKAN.Program (null) - The remote server returned an error: (404) Not Found.
2430 [1] FATAL CKAN.NetKAN.Program (null) -   at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationTokencancellationToken) [0x00146] in <e8eb3d7a311640f484845e45cbec8973>:0 
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <e8eb3d7a311640f484845e45cbec8973>:0 
at System.Net.HttpWebRequest.GetResponse () [0x00016] in <e8eb3d7a311640f484845e45cbec8973>:0 
at CKAN.NetKAN.Sources.Curse.CurseApi.ResolveRedirect (System.Uri url) [0x00025] in <fd6408a50cda4dc0acff3f19b62dcdb8>:0 
at CKAN.NetKAN.Sources.Curse.CurseFile.GetDownloadUrl () [0x00049] in <fd6408a50cda4dc0acff3f19b62dcdb8>:0 
at CKAN.NetKAN.Sources.Curse.CurseFile.GetFileVersion () [0x0000f] in <fd6408a50cda4dc0acff3f19b62dcdb8>:0 
at CKAN.NetKAN.Transformers.CurseTransformer.TransformOne (Newtonsoft.Json.Linq.JObject json, CKAN.NetKAN.Sources.Curse.CurseMod curseMod, CKAN.NetKAN.Sources.Curse.CurseFile latestVersion) [0x00011] in <fd6408a50cda4dc0acff3f19b62dcdb8>:0 
at CKAN.NetKAN.Transformers.CurseTransformer+<Transform>d__5.MoveNext () [0x00183] in <fd6408a50cda4dc0acff3f19b62dcdb8>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x0006f] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0 
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x00038] in <35ad2ebb203f4577b22a9d30eca3ec1f>:0
at CKAN.NetKAN.Processors.Inflator.Inflate (System.String filename, CKAN.NetKAN.Model.Metadata netkan, System.Nullable`1[T] releases) [0x000bf] in <fd6408a50cda4dc0acff3f19b62dcdb8>:0
at CKAN.NetKAN.Program.Main (System.String[] args) [0x00150] in <fd6408a50cda4dc0acff3f19b62dcdb8>:0

DasSkelett commented 5 years ago

Did they shut down the old API?

HebaruSan commented 5 years ago

https://github.com/KSP-CKAN/NetKAN/pull/6608#issuecomment-401098413

The system has many legacy URLs in the cache, and continues to support them, but for new projects you must use the current valid CurseForge URL to access the project data through the widget.

I'm guessing the cache got cleared. Can you try updating one of them to the new format? KSP-CKAN/CKAN#2464

DasSkelett commented 5 years ago

I'm guessing the cache got cleared. Can you try updating one of them to the new format? KSP-CKAN/CKAN#2464

You mean replacing the number id in the kref with the mod name (e.g. 283006 with discordrp)? That didn't work, same error for the different url :/

514 [1] INFO CKAN.NetKAN.Sources.Curse.CurseApi (null) - Calling https://api.cfwidget.com/kerbal/ksp-mods/discordrp
514 [1] DEBUG CKAN.Net (null) - About to download https://api.cfwidget.com/kerbal/ksp-mods/discordrp
2717 [1] DEBUG CKAN.NetKAN.Processors.Inflator (null) - Deleting downloads for failed inflation
2717 [1] FATAL CKAN.NetKAN.Program (null) - The remote server returned an error: (404) Not Found.
HebaruSan commented 5 years ago

It's not failing on that URL. With some extra logging added:

668 [1] INFO CKAN.NetKAN.Sources.Curse.CurseApi (null) - Calling https://api.cfwidget.com/project/283006
669 [1] DEBUG CKAN.Net (null) - About to download https://api.cfwidget.com/project/283006
8016 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseMod (null) - Getting page URL of 283006
8016 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseMod (null) - Resolving page URL of 283006
8016 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseApi (null) - Resolving redirects of https://kerbal.curseforge.com/projects/283006
8016 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseApi (null) - Attempting to get response
8485 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseApi (null) - Got response
8486 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseApi (null) - Location header is https://www.curseforge.com/kerbal/ksp-mods/discordrp
8952 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseApi (null) - Resolved redirects to https://www.curseforge.com/kerbal/ksp-mods/discordrp
8953 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseMod (null) - Resolved to https://www.curseforge.com/kerbal/ksp-mods/discordrp
8953 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseMod (null) - Getting page URL of 283006
8953 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseMod (null) - Getting page URL of 283006
8953 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseMod (null) - Getting page URL of 283006
8953 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseMod (null) - Getting page URL of 283006
8957 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseApi (null) - Resolving redirects of https://www.curseforge.com/kerbal/ksp-mods/discordrp/files/2532403/download
8957 [1] DEBUG CKAN.NetKAN.Sources.Curse.CurseApi (null) - Attempting to get response
9250 [1] DEBUG CKAN.NetKAN.Processors.Inflator (null) - Deleting downloads for failed inflation
9250 [1] FATAL CKAN.NetKAN.Program (null) - The remote server returned an error: (404) Not Found.
9257 [1] FATAL CKAN.NetKAN.Program (null) -   at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <e8eb3d7a311640f484845e45cbec8973>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <e8eb3d7a311640f484845e45cbec8973>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <e8eb3d7a311640f484845e45cbec8973>:0 
  at CKAN.NetKAN.Sources.Curse.CurseApi.ResolveRedirect (System.Uri url) [0x0004b] in <eea182af7bbc49d698229a95a7ee93f6>:0 
  at CKAN.NetKAN.Sources.Curse.CurseFile.GetDownloadUrl () [0x00049] in <eea182af7bbc49d698229a95a7ee93f6>:0 
  at CKAN.NetKAN.Sources.Curse.CurseFile.GetFileVersion () [0x0000f] in <eea182af7bbc49d698229a95a7ee93f6>:0 
  at CKAN.NetKAN.Transformers.CurseTransformer.TransformOne (Newtonsoft.Json.Linq.JObject json, CKAN.NetKAN.Sources.Curse.CurseMod curseMod, CKAN.NetKAN.Sources.Curse.CurseFile latestVersion) [0x00011] in <eea182af7bbc49d698229a95a7ee93f6>:0

https://www.curseforge.com/kerbal/ksp-mods/discordrp/files/2532403/download fails in my browser just like it does in netkan.exe. So something is wrong with the download URLs in the Curse API.

HebaruSan commented 5 years ago

Project page: https://www.curseforge.com/kerbal/ksp-mods/discordrp

Download link: https://www.curseforge.com/kerbal/ksp-mods/discordrp/download (Leads to "Your Download will begin in 5 seconds.")

@shrink, are there any recent updates about the Curse API situation?

shrink commented 5 years ago

@HebaruSan This is my last week as a Curse employee: the service is in limbo at this point. I've been looking for someone Curse-adjacent to take over it but nothing is forthcoming, and CurseForge themselves don't seem to be motivated to take it on. I'll keep the service running myself independent of my employment at Curse but I will no longer be actively managing/maintaining it, so any issues that crop up will, unfortunately, go unresolved. The service has been relatively stable and I assume that it'll continue to function fine but... if I were relying on it, I'd consider at this point whether you want to continue to rely on the service and what alternatives you have.

Hopefully CurseForge will launch their own API at some point but I think I've been saying that for years at this point and we are (as far as I know) no closer to that happening. Sorry that there isn't happier news.

HebaruSan commented 5 years ago

Hey @ihsoft, you have several mods using metanetkans and hosted on Curse:

New releases on Curse won't be auto-indexed anymore because the Curse API has started returning invalid download URLs. I'll be looking to migrate other mods to alternate download hosts, but I can't do that for metanetkans since the metadata is hosted in your repo.

techman83 commented 5 years ago

Thanks for all your efforts @shrink !

ihsoft commented 5 years ago

@HebaruSan Will CKAN stop supporting CurseForge going forward or is it a temporary measure until they fixed their API? I can ping them and see if it can get fixed soon.

HebaruSan commented 5 years ago

@ihsoft, there are no immediate plans to drop Curseforge, but it isn't working currently, and at some point it will be misleading for us to still mention it in our documentation if it never works again. If you can get it fixed, that would be fantastic!

ihsoft commented 5 years ago

@HebaruSan, Could you please help me understanding what exactly is not working right? I did a quick test with KIS project. Requested JSON with: https://api.cfwidget.com/project/228561. Extracted the very first item from files:

    {
      "id": 2718071,
      "url": "https://www.curseforge.com/kerbal/ksp-mods/kerbal-inventory-system-kis/download/2718071",
      "name": "KIS_v1.22.zip",
      "type": "release",
      "version": "1.7.1",
      "filesize": "25129124",
      "versions": [
        "1.7.1"
      ],
      "downloads": 9566,
      "uploaded_at": "2019-05-31T08:15:23Z"
    },

Then attempted to download: https://www.curseforge.com/kerbal/ksp-mods/kerbal-inventory-system-kis/download/2718071. Got HTTP codes:

image

The last HTTP 200 gives KIS v1.22 release archive. It works just fine to me (opening using WinRAR).

HebaruSan commented 5 years ago

Sure, the problem is that that URL doesn't link to the file. It links to an HTML page that uses Javascript to start a download in your browser. CKAN isn't a browser and doesn't load and run HTML or Javascript from the "loading pages" of remote servers.

HebaruSan commented 5 years ago

Hmm, this one does work though (generated by appending /file to files[whatever].url (as shown above):

https://www.curseforge.com/kerbal/ksp-mods/kerbal-inventory-system-kis/download/2718071/file

Maybe we can still salvage this with some code changes in Netkan...

HebaruSan commented 5 years ago

@ihsoft, you can probably forget about changing your mods; we will find out for sure during the next bot pass, sometime in the next 2-8 hours. Curse lives to fight another day.

EDIT: Confirmed, it's working again. Sorry for the false alarm.