search

KSP-CKAN / CKAN

The Comprehensive Kerbal Archive Network
https://forum.kerbalspaceprogram.com/index.php?/topic/197082-*
Other
1.93k stars 348 forks source link

Indexing from Curse #1599

Closed Dazpoet closed 7 years ago

Dazpoet commented 8 years ago

@DuoDex pointed out on irc that we might be able to use https://widget.mcf.li/ to hook into Curse and grab data for inflating .netkans pointed at Curse directly rather than manually making every single .ckan file.

This was also, allegedly, ok'd by a Curse-person named JadeCat who came by the channel to ask what was preventing us from indexing Curse-hosted mods.

As for the technical how-to I have no idea but I'm creating this issue so we can track the discussion. Sounds like a really sweet deal if someone can get it working :+1:

DuoDex commented 8 years ago

So, after talking with most of #ckan and relaying discussion from the Curse rep @jadedcat, it was decided that an effort would be made to add Curse autoindexing using the site linked above's API.

We can download directly from Curse, that's never been an issue, and all we need to need to do is a) figure out how to get a bot to index using the API, which I think I ping @zengei for and b) do other things to make it all work.

dbent commented 8 years ago

Using that API is not viable in its current form and I'm going to be very detailed here as there seems to have confusion in IRC as to why.

Let's look at the data returned for Kerbal Attachment System: https://widget.mcf.li/ksp-mods/kerbal/223900-kerbal-attachment-system-kas.json (below)

We need a download URL that points to the ZIP file, the url properties returned above are to an HTML document, which then subsequently redirects the user's browser to the actual ZIP file.

The actual ZIP file is hosted on Curse's CDN with a URL like so (from KAS's .ckan): http://addons.curse.cursecdn.com/files/2249/712/KAS_v0.5.4.zip.

Now let's see if we can somehow generate this URL from the data provided above. Looking at the URL we reduce it to the template: http://addons.curse.cursecdn.com/files/{A}/{B}/{C}.

So we need to figure out what {A}, {B}, and {C} are. It looks like {A} is the first 4-digits of the file id and {B} is the last 3-digits of the id. But what if id is 8-digits long? Is {A} the first 5-digits? Is {B} the last 4-digits? Or maybe another segment is added to the path? What if id is 6-digits long? Is {A} 3-digits or is {B} 2-digits? What guarantee do we have that whatever we're doing is actually the right thing?

But worse yet, {C} is nowhere in the data at all. It looks like the name property but it's not, the name property is just an arbitrary string the author chooses. For example for MechJeb they look like: MechJeb 2.5.5 for KSP 1.0.5, MechJeb 2.5.3 for KSP 1.0.4, etc. when the actual file is MechJeb2-2.5.5.0.zip, MechJeb2-2.5.3.0.zip, etc.

If we want the actual download URL, we'd have to try to scrape it from the HTML page given in the url property, at which point we might as well just scrape to get the latest version as well and not even use the above API.

Adding @pjf since apparently he had some insight into this that I'm missing. Also, I'm zengei from IRC, not whomever you at-mentioned.

{ "authors" : [ "kospy44" ],
  "category" : "Physics",
  "created_at" : "2014-08-31T16:04:42+0000",
  "download" : { "created_at" : "2015-11-11T17:00:33+0000",
      "downloads" : 130177,
      "id" : 2266204,
      "name" : "KAS v0.5.5",
      "type" : "release",
      "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2266204",
      "version" : "1.0.5"
    },
  "downloads" : { "monthly" : 36760,
      "total" : 730063
    },
  "favorites" : 1095,
  "files" : { "2213185" : { "created_at" : "2014-08-31T20:46:34+0000",
          "downloads" : 41069,
          "id" : 2213185,
          "name" : "KAS_0.4.8.zip",
          "type" : "release",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2213185",
          "version" : "0.24.2"
        },
      "2216138" : { "created_at" : "2014-10-08T18:15:00+0000",
          "downloads" : 113361,
          "id" : 2216138,
          "name" : "KAS v0.4.9",
          "type" : "release",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2216138",
          "version" : "0.25"
        },
      "2222185" : { "created_at" : "2014-12-17T20:59:43+0000",
          "downloads" : 183550,
          "id" : 2222185,
          "name" : "KAS v0.4.10",
          "type" : "release",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2222185",
          "version" : "0.90"
        },
      "2238224" : { "created_at" : "2015-05-14T16:06:12+0000",
          "downloads" : 1665,
          "id" : 2238224,
          "name" : "KAS v0.5.0",
          "type" : "release",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2238224",
          "version" : "1.0.2"
        },
      "2238230" : { "created_at" : "2015-05-14T17:51:16+0000",
          "downloads" : 901,
          "id" : 2238230,
          "name" : "KAS v0.4 old parts for KAS v0.5 (optional)",
          "type" : "beta",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2238230",
          "version" : "1.0.2"
        },
      "2238254" : { "created_at" : "2015-05-14T20:14:53+0000",
          "downloads" : 34163,
          "id" : 2238254,
          "name" : "KAS v0.5.1",
          "type" : "release",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2238254",
          "version" : "1.0.2"
        },
      "2240844" : { "created_at" : "2015-05-31T21:00:53+0000",
          "downloads" : 40305,
          "id" : 2240844,
          "name" : "KAS v0.5.2",
          "type" : "release",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2240844",
          "version" : "1.0.2"
        },
      "2246546" : { "created_at" : "2015-07-11T18:03:14+0000",
          "downloads" : 42588,
          "id" : 2246546,
          "name" : "KAS v0.5.3",
          "type" : "release",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2246546",
          "version" : "1.0.4"
        },
      "2249712" : { "created_at" : "2015-07-29T20:48:11+0000",
          "downloads" : 142687,
          "id" : 2249712,
          "name" : "KAS v0.5.4",
          "type" : "release",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2249712",
          "version" : "1.0.4"
        },
      "2266204" : { "created_at" : "2015-11-11T17:00:33+0000",
          "downloads" : 130177,
          "id" : 2266204,
          "name" : "KAS v0.5.5",
          "type" : "release",
          "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2266204",
          "version" : "1.0.5"
        }
    },
  "game" : "Kerbal Space Program",
  "license" : "Custom License",
  "likes" : 51,
  "project_url" : "http://www.curseforge.com/projects/223900/",
  "release_type" : "Release",
  "thumbnail" : "https://media-curse.cursecdn.com/attachments/147/665/3977e3eb526c6b5e5ede942ccbd8ff38.png",
  "title" : "Kerbal Attachment System (KAS)",
  "updated_at" : "2015-11-11T17:48:42+0000",
  "url" : "http://www.curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas",
  "versions" : { "0.24.2" : [ { "created_at" : "2014-08-31T20:46:34+0000",
            "downloads" : 41069,
            "id" : 2213185,
            "name" : "KAS_0.4.8.zip",
            "type" : "release",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2213185",
            "version" : "0.24.2"
          } ],
      "0.25" : [ { "created_at" : "2014-10-08T18:15:00+0000",
            "downloads" : 113361,
            "id" : 2216138,
            "name" : "KAS v0.4.9",
            "type" : "release",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2216138",
            "version" : "0.25"
          } ],
      "0.90" : [ { "created_at" : "2014-12-17T20:59:43+0000",
            "downloads" : 183550,
            "id" : 2222185,
            "name" : "KAS v0.4.10",
            "type" : "release",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2222185",
            "version" : "0.90"
          } ],
      "1.0.2" : [ { "created_at" : "2015-05-31T21:00:53+0000",
            "downloads" : 40305,
            "id" : 2240844,
            "name" : "KAS v0.5.2",
            "type" : "release",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2240844",
            "version" : "1.0.2"
          },
          { "created_at" : "2015-05-14T20:14:53+0000",
            "downloads" : 34163,
            "id" : 2238254,
            "name" : "KAS v0.5.1",
            "type" : "release",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2238254",
            "version" : "1.0.2"
          },
          { "created_at" : "2015-05-14T17:51:16+0000",
            "downloads" : 901,
            "id" : 2238230,
            "name" : "KAS v0.4 old parts for KAS v0.5 (optional)",
            "type" : "beta",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2238230",
            "version" : "1.0.2"
          },
          { "created_at" : "2015-05-14T16:06:12+0000",
            "downloads" : 1665,
            "id" : 2238224,
            "name" : "KAS v0.5.0",
            "type" : "release",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2238224",
            "version" : "1.0.2"
          }
        ],
      "1.0.4" : [ { "created_at" : "2015-07-29T20:48:11+0000",
            "downloads" : 142687,
            "id" : 2249712,
            "name" : "KAS v0.5.4",
            "type" : "release",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2249712",
            "version" : "1.0.4"
          },
          { "created_at" : "2015-07-11T18:03:14+0000",
            "downloads" : 42588,
            "id" : 2246546,
            "name" : "KAS v0.5.3",
            "type" : "release",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2246546",
            "version" : "1.0.4"
          }
        ],
      "1.0.5" : [ { "created_at" : "2015-11-11T17:00:33+0000",
            "downloads" : 130177,
            "id" : 2266204,
            "name" : "KAS v0.5.5",
            "type" : "release",
            "url" : "http://curse.com/ksp-mods/kerbal/223900-kerbal-attachment-system-kas/2266204",
            "version" : "1.0.5"
          } ]
    }
}
DuoDex commented 8 years ago

However, I was led to believe that the actual download would not be the issue by both jaded and pjf.

I was also linked this, which is also in Python and has the capability of downloading from the Curse CDN.

Finally, I may seem like I'm talking out of my rear. This is because I am, pretty much, so please don't expect too much from me!

Dazpoet commented 8 years ago

It seems that at some point there was a client able to use mod-urls to autoupdate mods from curse, it was named TinkerTime but based on this issue I'm going to assume it was not a painless endevour.

airminer commented 8 years ago

I think I found a mostly painless way to implement web scraping for curse, but it would involve an additional dependency on HtmlAgilityPack.

airminer commented 8 years ago

OK, in the end I found a way to do it with just https://widget.mcf.li/, without any additional dependencies. See PR https://github.com/KSP-CKAN/CKAN/pull/1608

Olympic1 commented 7 years ago

Added in v1.20.0