Curse support

[BobPalmer]

I didn't see it floating around, but given the request for folks to use repos other than the official one, you should really consider Curse support if you wish for this project to achieve a more universal adoption rate.

[pjf]

I haven't checked Curse's terms and conditions, but I don't think they provide an API, nor do I believe they'd be happy with us scraping their website. (I would be very happy to be corrected here!)

We can download from anywhere—including curse—if that's what's listed in the control metadata, but I'm not sure it would be easy to scrape curse for when mods have been updated in a friendly manner.

I suspect that AVC has the same issue here (but again, I'd be very happy to be corrected there, too!).

[AlexanderDzhoganov]

I vote that we don't do this even if we can, Curse is a for-profit website with a bad reputation within the general KSP community, very far from the spirit of CKAN. In my view, we want to discourage mod authors to post only to Curse and do at least a KerbalStuff release. I don't have anything against Curse.com personally (I use it for my mod), it's just what I have observed is the norm.

[pjf]

I'm totally fine with for-profit websites, and I'd rather us chew curse's bandwidth than KS's bandwidth. But as a much more closed system, I doubt they have any sort of API that will let us integrate cleanly. :/

I imagine most CKAN authors will favour releasing to KS simply because it makes their lives easier.

[BobPalmer]

@AlexanderDzhoganov reality is though it is the official repo, regardless of personal opinion. And to have CKAN either discourage or downplay it is more than a little bit short sighted. FYI - they do have an API, though what I saw was on the publishing side. I also think the whole 'bad rep' bit is more than a bit overblown by a very vocal minority.

Conversely, if CKAN were to heavily push KerbalStuff as the preferred repo, it would pretty much drive me not to participate. I'd rather see this as something that is truly hosting-neutral and not embroiled in the whole 'omg I hatez curse' political BS. @pjf - there are always Curse folks in the #KSPModders channel, Kaelten (sp?) is the developer dude, so a PM would not go amiss.

[AlexanderDzhoganov]

@BobPalmer Thank you for the response. You make a good point, so I withdraw my previous statement. Still, this is @pjf's decision to make and I will support it, whatever it is.

[pjf]

I'm really appreciating the discussion here, and while I don't think matters should be my decision, I'm more than happy to contribute my views. :)

So, I think we might have a bit of confusion here. The CKAN is an index of metadata, which allows clients to download and install mods using pre-defined rules and relationships. It doesn't support one site or another per se; provided one can write the metadata to describe how a mod should downloaded and installed, and provided the license on the mod permits this to occur, the CKAN is site agnostic. There's no requirement to host your project on KerbalStuff, Github, Curse, Geocities, or anywhere else.

KerbalStuff provides a very rich API which allows us to get a wealth of information about mods; this means that bots are able to generate almost complete metadata given only a few pieces of information. Github provides a more generic API that simply can't tell us things like supported KSP version, but which still lets us pick up versions and file info. Consequently, both of these websites make it easier for mods to transition into the CKAN, and for new releases to be indexed quickly, because they shift the burden of metadata generation from humans to bots.

If Curse, or any other website, has an API that provides us useful information, then I want us to use that information provided their terms of use allow us to do so. If their API does not allow us to use it, or doesn't provide us with information we need¹, then Curse-hosted mods can still be installed via the CKAN, it's just that they need someone to write the metadata for them first, and that metadata will need to be updated when there are new releases. That "someone" is likely to be the mod author, or (in the case where the author is unwilling or unable) will likely be a fan of the mod.

I'm not at all familiar with Curse; there is mention of an API, but the docs I've found have been minimal. I'm really glad to hear they have folks on IRC, and I'll definitely approach them to see if there's anything we can use. I'll also tag @curseforge here, in case they're at all active on github.

So just to be very clear here, I would like to support Curse if they have end-points we can use. It would be a huge boon to the CKAN to make it easier for existing authors to generate the metadata. I don't want to push one host over another. Ultimately, I want authors to be the ones who are ultimately managing their metadata and mods, no matter where they might be released.

The fact that it's easier to get started indexing mods from KerbalStuff is purely because their API is both rich and unrestricted, and I hope this support will be given to any service that's equally as easy to use. Our current support framework is due to technical ease, not policy decisions.

As for decisions being mine, I'm very big on community discussion, and even bigger on doing whatever makes the world a better place to be in.

Many, many thanks!

~ Paul

¹ At the very least we need to be able to detect a release, its version, and where to download it.

[BobPalmer]

Thanks :) Just wanted to make sure this project has the best chance of success.

[Almagnus1]

I've read through the issue, and would like to clarify a couple of points:

Are we referring to automated updating from Curse?

If a mod author has a mod on Curse, can they still manually create the necessary files to have CKAN work with the mod?

[AlexanderDzhoganov]

Hey, @Almagnus1 The CKAN client can download from anywhere so manually adding Curse- hosted mods works. What doesn't work is the NetKAN utility which can automatically generate metadata on new releases.

[TeddyDD]

Well, I'm working on a simple tool that let us to automate adding mods from curse (similar to netkan) It can fetch releases urls, version and ksp versions. At the beginning of Christmas I'll try to show you first test version :) NO WARRANTY! I'm an artist, not a programmer so I do not know if this will work as I wanna.

[AlexanderDzhoganov]

@TeddyDD, have you worked out any terms with Curse, because the ToS of their website explicitly forbids any scraping?

[Ippo343]

If I am reading the ToS correctly, you could in theory do it as long as you introduce an artificial delay between requests and avoid parallel downloads, so that you don't cause more requests than a human could reasonably do.

I'm not suggesting that we exploit this loophole, though: we shouldn't be crippling our client to comply to their ToS.

[AlexanderDzhoganov]

@Ippo343, considering how simple it is to rate- limit requests I would interpret this as disallowing bots completely.

[Ippo343]

Of course: as I said, while we could technically comply to the ToS, I don't think we should do it and just not support it.

[TeddyDD]

@AlexanderDzhoganov I haven't idea about this :neutral_face: I'll dig into this.

[TeddyDD]

Well, I see two possible interpretations: Or prohibit anything that could harm their website (and harvesting metadata only would use one request message per mod)

You may not connect to or use the Websites in any way not expressly permitted by this Terms of Service. Without limiting the foregoing, you agree that you will not (i) institute, assist, or become involved in any type of attack, including without limitation denial of service attacks, upon the Websites or otherwise attempt to disrupt the Websites or any other person's use of the Websites; (ii) use or launch any automated system, including without limitation, "robots," "spiders," "offline readers," etc., that accesses the Websites in a manner that sends more request messages to the Curse servers in a given period of time

Or they prohibit any use of curse that is not ordinary web browsing and then we should delete all hand crafted curse mods :cry:

copy or distribute any part of the Websites in any medium without Curse's prior written authorization

[vosechu]

When I installed CKAN today it found several modules from Curse; KAS being an example. Has this issue been fixed? Or has it been worked around in some other way?

[hakan42]

My best guess is that you encountered addons that were both on Curse and somewhere else (kerbalstuff comes to mind, or github) and were thusly indexed.

Either that or someone wrote a hand-crafted ckan for them.

[politas]

Does Curseforge have the same problem Curse does? I'm not sure what the relationship is between them.

[pjf]

Curse don't have an API that we can use, so we can't really do this in terms of automatic indexing. Marking this as closed for now; we can (and do) manually index curse-only mods, but that requires a volunteer or the mod author making sure their info is being kept up to date.