Better organization for KAG releases

[asumagic]

Release throughput is rather terrible partly because contributors (especially me) have finite motivation and time to help with releases.

Some people have been stepping up recently to help (e.g. Mazey) but the release process is currently way sub-optimal since core contributors are loaded with work that takes away time that could be spent on things others mostly can't do (cough staging release cough).

I think this is at least partly due to rather terrible organization, so this thread is written with the hope of creating a more robust procedure for KAG releases that reduces reliance on the very few "all-powerful" contributors and sparking some discussion on what could be improved. The current procedure is barely documented and was much better suited for a time where the game was not as community-driven and when core developers had usually more time to provide.

Required work on releases

There are a couple things I generally need to do which represents most of the time I need to spend when pushing KAG releases (engine work and debugging aside):

1. Reviewing PRs. Code review is hard and annoying but it ultimately reduces delays caused by finding bugs late in testing. A few people have been (very helpfully) reviewing PRs but the massive number of PRs makes it difficult to catch up and I still end up reviewing most of them.
2. Writing changelogs. We do have automated changelogs in the CI/CD, but a. nearly nobody bothers writing good, descriptive commit messages as the [added]/[dev]/etc. convention is not enforced, b. they often don't properly represent all changes and c. they don't attribute the changes to a person, so we have to manually add that.
3. Writing posts: Discord, Steam, etc., which reuses the same changelogs mentioned before, but formatting, writing down a summary and checking the syntax is also time-consuming. Steam requires creating assets for announcement posts, which if helpful for visibility is also annoyingly time consuming.
4. Testing. Organizing test sessions, following a procedure (e.g. for mapcycle testing), scanning through merged PRs to see what needs to be tested, setting up test servers, etc. is overall time consuming.
5. Catching up with stuff that is not written in TODO lists but really should, and that sometimes go forgotten - e.g. Patreon heads.
6. Decide what to merge and what not to. This is more difficult than it appears because first, there is no consensus for some of the changes and little discussion (although CTF rebuild had a good approach to gathering feedback, it is not the norm and its changes are still controversial to some), and second, some PRs impactful on gameplay are published without prior discussion, and not merging PRs is a frustrating mess for everybody involved.
7. Keeping staging updated and solve issues arising from changes that don't affect release. I have been trying to create staging builds ahead of KAG releases so that I don't get spammed with "staging pls:((" for days/weeks but it's sometimes lengthy.
8. Monitoring and hotpatching in case something goes wrong. Surprisingly, this isn't that common or that time consuming to deal with.

Things that should be done

The first 6 points have a lot of room for optimization. There's a few things we might want to do:

1. Requiring checklists on PRs, e.g. "was this tested on net?". Sometimes this goes forgotten and passes code review and subsequently wastes time during testing.
2. Further encouraging code review.
3. Document a proper workflow for testing other's PRs easily (to catch more sneaky bugs before they hit the test branch).
4. If code review is blocked for non-contributors (which I have heard someone say), it should be allowed. While it wouldn't mean lowering the level of scrutiny by core contributors significantly (though tbh that would depend on the experience of reviewers), getting more people to read and understand changes would almost certainly save time catching bugs early and improve code quality.
5. Streamlining a way to write changelogs and social posts, and offload their writing to other people. Ideally, this could be done incrementally. Documenting what Steam post assets are required is also important, and it depends on the type of update.
6. Streamlining a test procedure and offload organizing testing sessions to other people. Also, test a few common mods for breakage, as those could indicate engine bugs (e.g. Shiprekt issues from last build) - although avoiding script breakage is not a priority.
7. There were talks long ago about how organizing discussion on gameplay changes should be done. I think the main takeaway is that right now Discord is not well suited for long wided discussions that we want to keep track of, and GitHub has less visibility and less people bothering to discuss on there. Should anything be changed there?
8. ... merging staging, which has admittedly always been a priority, but the lack of which has increasingly been a pain point. The amount of git history fuckery is slowly increasing, etc. Creating a branch as we already have release/test/dev/etc config may require significant CI/CD work, but people who are able to help me with this are even more busy than I am.

This is mostly a dump of my thoughts so far on how the development process could be improved, so that there is at least a trace of it.

[Mazey]

I think short-term it would be great to at least get something happening so people keep playing the game and contributors besides mugg may get some motivation. I'm not going to reply to any specific points directly but I'll just state what I think should happen ASAP.

To start I will state that I am always available to fully fulfill all social needs: promo materials, Steam + Discord announcements/events, etc. You can completely forget about those parts, as merged commits and "being involved" is enough for me to know what to work on.

What I would like to see short-term is the merging of simple commits (that possibly have already been reviewed) that do not change the game balance heavily, as well as more minor bug fixes and QoL changes. This doesn't have to be entirely production-ready, but I do think it's safe to say all commits have at least been tested locally to work, so if we simply merge these into test, even if test breaks we can fix it.

This is far easier than full code reviews & merging commits locally to test them online. Crashes or bugs can be fixed far easier that way; I would open a server on test in a few minutes and people could go to work if anything is broken. I know typically it has been @AsuMagic @eps0003 @Vam-Jam and occasionally me reviewing PRs, but if people can easily "test" 20-30 PRs by just updating the test client I don't doubt we could see fixes from @bunniewormy @mugg91 or @GoldenGuy - people who are willing to contribute and have PRs ready to merge, and could fix issues that may arise on their own PRs or each other's.

[asumagic]

Also when it comes to changelogs -- it might be good to mention people who reviewed the PRs.