Add API for submitting change requests to the Index

[janbrohl]

While it currently is possible to request changes to the index there is no simple mostly automated workflow for that yet.

Currently there are 2 ways to request changes - indirectly via KSP-CKAN/NetKAN:

• You can submit issues which is simple to implement via GitHub API but all changes have to be reviewed by humans and need to be integrated manually.
• You can create pull requests which is quite complex (too complex for normal users) using GitHub API only or requires several steps needing git installed on the user's machine, some git-knowledge of the user, usage of GitHub API. This is much less work for the reviewers.

I think there should be a third option: Submitting change requests via a Web-API (REST + JSON ?) with simple authentication/authorization - this would make it possible for mod authors to include submitting current metadata automatically right into their release-process (eg with a simple command-line tool) - simplifying their workflow. This would also simplify submitting via special metadata-helpers like my Metadata-Webtool - simplifying workflow for new users. (This API could be interesting for the people of @KSP-SpaceDock ,too. )

A rather hacky proof of concept is available on https://github.com/janbrohl/NetKAN-API

[techman83]

@janbrohl can you provide a bit more detail about the problem you're trying to solve? The workflow is already pretty automated, to the point that there is no more hand generated metadata in CKAN-meta. It has all been re-generated using netkan.

GitHub already has an API with Authentication, something client side in JS would save us needing to worry about authentication and abuse of an API. I'm a fan of not re-inventing wheels.

[janbrohl]

I was asked to submit PRs instead of just issues with metadata-webtool. This is not trivial for me and the solution would be creating a library/API for this which is not constrained to just my metadata-webtool.

I am not git-savy enough to do this just by calling the GiHub API alone which would also require my app to request the rights of creating and editing repositories in the user's name as far as I understand, (Not good from a security perspective) combine API calls and local git on a server - basically requiring the same rights (would not host that myself), or I would have to implement my own auth and create the PRs from a bot account - like with janbrohl/NetKAN-API - (would not host that myself).

Having an API for update (request) submissions would not only benefit my webtool but also help others simplify/speed up their workflow, lessen load on the netkan indexer and possibly making the index a bit more up to date if it is possible to just update entries on new releases.

[techman83]

I'm still not super across the benefit. What percentage of NetKANs would benefit from this workflow? Most are automatically indexed already and all spacedock entries are autoinflated upon indexing.

Something we do need is Webhook authorisation workflow, that way authors could register their repos against our webhooks and inflate on demand.

[janbrohl]

I would think that such an API would simplify implementation of the Webhook workflow.

Currently i cannot provide data supporting the claim that anyone would benefit from this. Nobody actually seems to use my matadata-webtool.

[DinCahill]

I think @janbrohl's intention is to make an easy UI for people to be able to submit and edit netkans (and maybe ckans?) without knowing how to use git, which would also be able to be integrated into Metadata-Webtool.

Metadata-Webtool is there so people can make netkans without knowing JSON, and possibly without reading the CKAN spec (or maybe partially reading it). I've only had a quick look at the tool myself, but the idea is sound, I think.

So, @janbrohl is proposing we do something similar for git/GitHub. I personally believe that the tools provided by GitHub nowadays are sufficient. You can use the the GitHub website itself, or GitHub Desktop on Mac and Windows.

I think that using software for the job that Metadata-Webtool performs is justified, but for GitHub, we're better off using the existing tools, and writing good quality, well-structured, to-the-point documentation for the whole process of going from nothing to an indexed mod, with a minimal amount of prerequisite knowledge. Writing this documentation is much less work, and upkeep, than writing more software.

On the Arch Linux wiki, they have an Installation Guide, which is the canonical resource for how to install Arch, but if there's anything you don't already understand, you've got to go off and learn it by yourself. There is also the Beginner's Guide, which takes the installation process, and frames it in a way for less experienced users, so they don't need to have a deep understanding of every little thing. This approach definitely helped me to get started easily, and then learn more later.

I understand that writing software is more fun than documentation, but it's always possible to go too far. I believe that good documentation, in this case, would be more effective, and achievable with less of our effort, than writing more software.

Please let me know if you have any retorts to my gut feelings :smile:

[politas]

There's a fair bit of rewriting that is needed for the "Submitting a mod to CKAN" pages, and using the Metadata-webtool will feature in that, I'm sure.

[politas]

Currently, the Metadata-webtool is generating a pretty good .netkan file, which should need fairly minimal checking by CKAN team members before processing. It then sends the user to the new Issue form in NetKAN, with the .netkan file attached and useful information. This is good, but a PR a la Space-Duck would be better. I'm not entirely sure how the Space-Duck submission process works. If we can work out a process to use existing GitHub APIs for people using the Metadata-webtool, that would of course be better than creating extra technical debt. Maybe a dedicated Github account in the same vein as SpaceDuck would be the way to do that, rather than trying to use the submitter's GitHub account?

The use case for this is indeed people who want to add a mod to CKAN without learning too much about JSON or the CKAN spec. From that perspective, it might actually be better to get it submitting Issues rather than PRs, so we aren't relying on the submitter to make tweaks. @KSP-CKAN/wranglers?

@janbrohl, do you have a way to import an existing netkan file for updates/corrections? That might be the logical next improvement to the tool.

[janbrohl]

Importing netkan files for editing is partly implemented but untested - currently install directives, non-standard fields and kref-related extra fields are dropped silently. (Help on this would be appreciated)

Spacedock seems to handle netkan submission (like NetKAN-API ) by submitting pull requests via a bot-account requiring an authenticated and authorized user (logged in mod maintainer)

This seems like the simplest way to implement simple netkan submissions apart from creating github-issues but it requires some way of authorization-checking to prevent spam if exposed in a way that fits metadata-webtool as this would be an API open for submitting without going through the form before.

[DinCahill]

Oh dear I fat fingered it.