KSP-CKAN / CKAN

The Comprehensive Kerbal Archive Network
https://forum.kerbalspaceprogram.com/index.php?/topic/197082-*
Other
1.96k stars 349 forks source link

The Future CKAN: repositories, releases, workflows, and support #807

Open pjf opened 9 years ago

pjf commented 9 years ago

I'm back working on the CKAN on a regular basis, and I've been looking at both my workflow, and the new user experience. There's a number of things I think we can improve, and I'd like to bounce some ideas off you all to hear your thoughts:

Fewer Repos

The KSP-CKAN project has a lot of repositories now. The original CKAN repo has been broken up into six separate repos: the core, NetKAN tools, GUI, command-line, support, and the original CKAN repo itself.

While I was the one who originally suggested splitting the repos, I'm finding it personally difficult to navigate the new structure. Each repo needs separate updating on the command line, and when it comes to issues tools like waffle have a harder time working with multiple repos. I'm certainly finding it harder to search, triage, and generally keep on top of issues.

My biggest motivation in splitting repos was to separate the CKAN core from the rest of the codebase. I'd like for the core to have its own releases (as ckan.dll), target the .NET 3.5 CLR so it can be loaded into Unity, and ideally have it available via NuGet so developers can easily write their own CKAN clients and tools.

I'm willing to be convinced otherwise, but I'm not sure splitting out the rest of the repos has gained enough benefit to outweigh the complexity this has caused. Consequently, I'd like to suggest the following:

I feel that we see a lot of the same questions being asked again and again. We have some great information in the wiki and FAQs, but users aren't really great at seeing them. A lot of the common questions can be solved with code, but we still need to answer those questions in the meantime.

Github + Waffle can do a reasonable job here, but it's not a fully fledged support system, with the ability for users to search for FAQs, for agents to prepare and use canned responses, the ability to track the state of tickets (eg: awaiting more information from user, etc), or straightforward ways to escalate to technical staff when required.

If we find that we're struggling with managing support due to the workflow issues presented by github, we might consider branching out to a dedicated support system like Freshdesk. We have a budget now, even if it's a very modest one, but it makes sense to use that on support options if we think they're going to be beneficial in the long term.

In this case, we may end up restricting the ability of non-contributors to create tickets in the CKAN repo, and provide a bridge by which support issues can be escalated to github issues. (Both freshdesk and github have APIs we can hook.) That means we can funnel all first contacts through our dedicated user-support system, and have a better chance of answering the common questions sooner.

Better homepage

Right now we don't have a real homepage, certainly not one that's flashy. I'm >70% confident we've got a sufficiently large mass of enthusiastic users that we can have a professional designer put something together for us, especially since I can think of two folks who have offered to do this off the top of my head. Even if it's just a simple landing page like Dancer with nice looking links to support, the wiki, downloads, and so on.

Better wiki

For the next two years, we're fully covered for all reasonable infrastructure expenses, thanks to AWS providing us with very generous sponsorship. We can and should spin up a gollum server, and set it up as wiki.ksp-ckan.org. Gollum is the same software that github uses for its native wikis, and we can even use the same repo underneath, so folks who edit on github will see those changes on wiki.ksp-ckan.org.

Supporting a wider ecosystem

To encourage a greater diversity of CKAN clients and applications, I'd like us to start making stand-alone ckan.dll releases, both on github and NuGet. These can target .NET 4.0 until we get around to making everything .NET 3.5 compatible.

Supporting easier install options

On both Mac and Linux the CKAN client requires a few extra dependencies. To streamline installation, I'd love to see us with apt-get, yum and homebrew solutions for both ksp-ckan and ksp-ckan-dev` for the user and dev environments respectively. I know Ippo has already made good headway with the apt-get/deb side of things.

Your feedback

I know that I'm very much re-entering the full CKAN development cycle after my break earlier this year, so feedback is very much appreciated. Especially if your name is @Ippo343, @AlexanderDzhoganov, @Dazpoet, or @hakan42, as I know you all have plenty of experience and insight to many of the ideas I've mentioned here.

Many thanks!

~ Paul

Wizarth commented 9 years ago

I agree with the user facing/interaction issues you've described. The developer-centric issues I have no opinion on.

TeddyDD commented 9 years ago

Even if it's just a simple landing page like Dancer with nice looking links to support, the wiki, downloads, and so on.

I don't think we need cgi scripts like Dancer when simple static website should be enough. I agree with the rest of the proposal :)

pjf commented 9 years ago

I don't think we need cgi scripts like Dancer when simple static website should be enough. I agree with the rest of the proposal :)

Sorry, I meant "in the style of" Dancer, or Bitelabs, or any other website that is essentially a single, nice looking page with links to resources. I'm not suggested we actually build our website using a particular technology, of the cloned cells of Kanye West turned into artisan meat products.

AlexanderDzhoganov commented 9 years ago

Great suggestions. I will help with whatever I can.

Dazpoet commented 9 years ago

As I mainly recide in the metadata and support places this will be mostly about that.

Fewer repos

I agree that the multiple repos pose something of a challenge and that some issues could be left hanging based on the lack of a good overview. I can see nothing bad come of some merging happening, acctually I'm already anticipating it!

As for this waffleboard thing I think a seperate one for metadata repositories would be amazing so people maintaining it can do another kind of triage there based on e.g. difficulty with tags like "Fix license", "Add dep/rec/sug" and "Manual overhaul".

As for removing CKAN-support: Yes, the idea behind it was superb but it is rather clear that it is currently splitting the userbase rather than funneling them into the right place.

Better support

Now this part I have a great interest in! I also feel that we keep seeing the same questions over and over again and very few people seem to locate the appropriate wikipages as to be able to help themselves. Many users not finding the wiki makes writing wikientries feel unrewarding and getting new contributors for the wiki might be hard purely based on percieved low impact, even though it could be one of the most helpful things a non-coder can do! It seems that often it is enough to point a user to the correct wikipage when they hop into irc to resolve many of their problems.

I wrote some about support just yesterday in https://github.com/KSP-CKAN/CKAN-support/issues/84 and I think that esspecially the idea of a basic guide to how we'd like to see error reports can help users help themselves and us. I've seen RoverDude maintain a steady demand for a picture of GameData before support is given for MKS/OKS and a similar "conditioning" of the the CKAN userbase with some basic questions we always ask and/or link to could probably go a long way. I'd love feedback in that issue btw.

To me the idea of escalation by contributors has a certain attraction since it might elevate some of the issueduplication we see today and could help funnel relevant information into a few topics rather than having to cross-reference multiple "perhaps related" issues like we do today. Could this perhaps also pave a way for contributors like myself who know very little of coding but like helping out in other ways?

Better wiki and homepage

I feel that this ties in a lot with the above discussion about support from my perspective. A better wiki could fix the issue of wikicontributions feeling less impactful. A new homepage would be an awesome place to prominently display to users how they can help us help them and also how they can help themselves. I love all of that! I have no idea how this gollum thing works but then again I also didn't have a clue about how github worked some time ago so :+1:

The rest of the things

I agree totally and look forward to helping in any way I can :)

Ippo343 commented 9 years ago

Hey Paul! So, here's my opinion:

Fewer Repos

I agree that merging / deleting some repos will make it easier to work on the project (I also hope it will come with a simpler build process). However, I think that netkan belongs to its own repository. Therefore, I would suggest to keep 3 repos, instead of two:

Better support

I agree on this one completely: our support system is lacking. Although I have to say that a big hole in our support strategy is the forum: many users post their issues on the forum and are not willing to bring the issue to github. Therefore, I think that the new system should allow the creation of a support ticket without registering an account, so that we can "politely encourage" the forum users to bring the issue there (and when I say "encourage", I mean "stop responding on the forum completely and just point them to the issue tracker").

Better homepage + better wiki

I agree. Nothing to add here :)

Wider ecosystem

Agreed here too. Although I wonder how much effort is really worth putting in supporting other clients when, as far as I know, there is no other client in development*.

Easier install options

As you pointed out, I have obtained some results with debian packaging. However, it's not really debian-compliant at the moment as I am not building the source but I'm starting with the precompiled binaries, which means it can never enter mainstream repositories in this state. It will be enough for a PPA (when I solve the remaining issues with the build), which will make it available through apt-get. At the same time, we could also provide an automated script to setup everything for less-experienced linux users: see my first attempt here.

Welcome back!

I'm very very glad that the patreon thing worked :) Now that you can devote some time regularly to the project, it should get up to full steam once again :)

Dazpoet commented 9 years ago

I will agree, a lot acctually, to what Ippo says about NetKAN being it's own repo, to me it would make sense if that repo also had the stuff currently in CKAN-meta as to collect all metadata related things in one place. The PRs and issues raised in NetKAN and CKAN-meta are distinctly different from those raised in other repos. The amount of users incorrectly posting error-reports into the metadata repos is marginal at most and the character of issues is often to add something or fix something metadata related rather than problems with the usage of CKAN itself. I think this is in part due to the name being NetKAN, users are looking for CKAN and don't really care much for anything not called CKAN when looking for support.

I don't think CKAN-core and/or CKAN-client repos would be best served by facing the onslaught of autogenerated PRs coming from KerbalStuff. At this moment NetKAN+CKAN-meta has about the same amount of PRs as CKAN has issues. Come KSP 1.0 updating metadata will probably be a decent sized chunk of work and well served by having its own space for handling that situation.

The effect on user support would also be marginal by using a 3 repo system rather than a 2 repo system. By some magic it seems that users mostly post metadata related requests in the right place and if something works, don't change it!

hakan42 commented 9 years ago

There is an advantage to keeping NetKAN and CKAN-meta separated though : it is much easier to set up the webhooks to build pull requests on Jenkins.

If we keep both in one repo, a netkan change would lead to a Jenkins build, which in turn would change ckans, which then in turn would trigger a (superfluous) jenkins build with the just created ckans yadda yadda yadda.

Of course it is possible to break up such circular builds with appropriate Jenkins job configuration but I am rather in favour of keeping them separate.

Ippo343 commented 9 years ago

Uh, just to clarify (I think I haven’t been very clear): when I said netkan I meant the netkan software, not the metadata.

I understood we were talking strictly about the code repos, not the metadata ones (which should remain separate, I think we all agree on that).

RichardLake commented 9 years ago

It may be due to having not developed since the repos was split much but I don't see the worth in having core separated out. With it you still have all the annoyances of multiple repos for only minor benefits. Unless XBuild has a undisclosed limitation the having projects in a solution doesn't mean they need to target the same .Net profile and the releases could be handled by having a separate repo similar to the nightlies.

pjf commented 9 years ago

@RichardLake : My goal in separating the core is that I'd very much like there to be CKAN clients and tools other than our own. It makes sense for there to be a separate repo that can be tracked for changes and PRs, and in which we can provide higher levels of quality assurance.

Having said that, you're right, we do get all the annoyances, and those annoyances are proving to be pretty great right now. Given that I wish to prioritise workflow, quality assurance, and accountability, I'm pretty sure merging all of CKAN-GUI, CKAN-Cmdline, CKAN-Core and CKAN would make sense. The last client release contained code that should never have got into the wild, and I would have more likely spotted it if our current build processes facilitated a comprehensive code review for each release. :/

Ippo343 commented 9 years ago

Regarding the support, why not redmine, which is both free an opensource?

RichardLake commented 9 years ago

If we are thinking of migrating issues off github keep in mind a number of companies offer free licenses to open source project such as Atlassian and Jetbrains.

pjf commented 9 years ago

@Ippo343 @RichardLake : I'd like to keep development and technical issues on github, but what it feels we really need is a dedicated support system. I'm very happy with a free offering, but the big needs I see are:

Support systems are very different beasts from ticketing systems, ideally they're all about answering the user's question without generating new tickets. Github makes for a great developing ticketing system, but a terrible support system. We really want to be able to reduce the amount of time spent fielding user support, especially when it comes to common questions and issues.

TL;DR: I'd love to keep dev/tech tickets on github, but deploy a dedicated support front-end for users, so they stop asking the same questions all the time.

pjf commented 9 years ago

@Ippo343: Since I'm hoping to begin The Great Merge in less than 48 hrs, you wouldn't be able to remind me of the tool you used to move tickets, could you?

pjf commented 9 years ago

@techman83 has pointed me to https://github.com/IQAndreas/github-issues-import , which looks very much like the output I saw for multi-ticket moves. :) Thank you! :)

Ippo343 commented 9 years ago

I was using this https://github-issue-mover.appspot.com

That is made by google, but I saw you found one already :)

-------- Messaggio originale --------
Da: Paul Fenwick
Data:08/05/2015 10:13 (GMT+01:00)
A: KSP-CKAN/CKAN
Cc: Michele
Oggetto: Re: [CKAN] The Future CKAN: repositories, releases, workflows, and support (#807)

@Ippo343: Since I'm hoping to begin The Great Merge in less than 48 hrs, you wouldn't be able to remind me of the tool you used to move tickets, could you?


Reply to this email directly or view it on GitHub: https://github.com/KSP-CKAN/CKAN/issues/807#issuecomment-100145409

BeatLink commented 1 year ago

Could CKAN be published on Steam as a dependency of KSP?

HebaruSan commented 1 year ago

@BeatLink please start a new issue if that interests you. This very old one is about some very different things.