search

MovingBlocks / Terasology

Terasology - open source voxel world
http://terasology.org
Apache License 2.0
3.68k stars 1.34k forks source link

Technology for workspace management and release tooling #4035

Open skaldarnar opened 4 years ago

skaldarnar commented 4 years ago

We've been on and off with discussing the need for more and better workspace tooling. This issue is intended to collect ideas and references regarding the topic to help us make an informed decision how to move forward.

Current Features

:construction: TODO: collect a list of features offered by groovyw ...

Requirements

:construction: TODO: figure out what we really need this for

References

Multi-Repo Tooling

Gradle

There are also Gradle Plugins for semver version management (either via git tag or version file), like https://github.com/ethauvin/semver-gradle.

Maybe this could be integrated into the terasology-module Gradle plugin in some way, so we would use Gradle to perform this task... I don't know how clean the code for that would, though, and how well this works with git interaction. And release plugins in general: https://github.com/researchgate/gradle-release

They usually work with version.properties or something similar, though.

CLI tooling

POCs

keturn commented 4 years ago

Okay, I wrote a proof-of-concept version of the module-release script in Kotlin.

Just for fun, I decided to deliberately not look at the node-gooey implementation beforehand, to see how much our initial approaches would differ.

Here it is: https://github.com/keturn/Terasology/blob/spike/gooky/logistics/src/main/kotlin/gooky.kt

I find it highly amusing. "How to turn a 7-command script in to 200 lines of Kotlin."

Many of those lines are thanks to JGit. I didn't find any better Kotlin wrapper of a git interface, and I wasn't sure if the rules of engagement allow calling out to command-line git or not.

There are some really obvious opportunities for refactoring to cut the line count down, but I wanted to see what it looked like using the interface as given.

(Disclaimer: this is also not production code. You'll want more safeguards and more ergonomic error-handling than this rough version for something that's going to push directly to your protected branches.)

keturn commented 4 years ago

Oh, for usage:

I did this as a subproject in the multi-repo, because it's essentially a requirement for working in the multi-repo.

Do gradlew :logistics:installDist to build, and then you'll end up with the standard java-application-wrapper-scripts in logistics/build/install/gooky/bin

in real life, of course, we'd move those to someplace easier to type.

Cervator commented 4 years ago

I'm trying to resist wall-of-texting various design notes and feedback here until I can finally get out of Jenkins-migration-hell. But a real quick design highlight for one of the main bits of the groovyw approach that led to most the awkwardness in code: I was trying to support all sorts of different parts of the workspace, not only modules, but also facades (like the server variant), libs (crash reporter, etc), meta-modules (for Blender models, raw music, and such), and even had plans to support the wiki repos associated with each thing, and possibly a separately checked out home for gh-pages if we were to go that way to host micro-sites for things.

I'm not saying that's great design - i tend to architect further than my arms can reach, fully acknowledged. But a thought to refactor that in Groovy or build something better in something else would be high on my list. Along with how to actually run it, since it is hard to beat 1) Have Java (+ Git); 2) Clone; 3) groovyw something; 4) Run the game (gradlew jar game or cram into IDE).

Lightweight and powerful utility is really the name of the game here. Maybe the initial groovyw fit that, but growing it into something bigger could make it increasingly awkward and lumbering.

skaldarnar commented 4 years ago

(Disclaimer: this is also not production code. You'll want more safeguards and more ergonomic error-handling than this rough version for something that's going to push directly to your protected branches.)

This disclaimer also applies to my node-gooey 😅

I like how this is integrated with the engine repo, this is a huuuge advantage of this solution. Just works with the tooling we have installed anyways. I see that we put different focus, and also had different assumptions (I think we agreed on not using develop branches anymore, so everything would happen on master).

I think I could befriend Clikt, but I'm missing a library to handle Semver stuff in your PoC - is there none, or didn't you bother using one?

keturn commented 4 years ago

I used gestalt's Version class.

I expect we'd find something if we went looking, but I guess I'm not sure what we'd be looking for that's different than what org.terasology.naming.Version currently provides.

And no branch switching? Ah, that's why node-gooey has so much less git stuff! It's not only because it has a more concise git interface.

skaldarnar commented 4 years ago

@Cervator I could sketch out how to scale out the JS solution to support different target "objects", reusing common code, etc. Basically just a "cleaner" way of implementing the pattern gooey <scope> <command> we have for groovyw, too.

keturn commented 4 years ago

Other consideration: performance.

A lot of these tools aren't super performance-sensitive (...although users with mega-workspaces might have some feelings on that), but order-of-magnitude differences are worth noting.

Cervator commented 4 years ago

Alright - finally got something I can contribute. I made it in the form of a PR since it works as-is and would be ready to go if completed: #4065 - in short is is just next gen existing stuff, with far fewer quirks and way more power.

I added a couple lines to the top post here. I'm mostly just addressing the CLI stuff, less so the release management, but if "GooeyCLI" is the pick I expect a handful of lines would add the release management (same GrGit as other stuff), with a handful more to automate interaction with GitHub releases (need to hit the API, prolly @Grab Octokit). Haven't done that yet since :time: plus wanting to see how the release management itself really turns out

Performance-wise I hobo-stopwatched my way to just shy of 2 seconds running the basic gw help command for what it is worth.

Extra comments beyond the fairly meaty engine PR:

As for the "current features" and "requirements" here - well. Fundamentally it is about utility and quality of life enhancements for anything in the workspace. I could probably write a wall of text on detailing that, but I ended up just writing some code instead. I think the existing groovyw module init omega really speaks for itself (now gw init omega - shorter!)

I'd like to go for this option, but I'm not mandating it. Trying to make it stand on its own and do well in a comparison. What do you think, @skaldarnar & @keturn?

Even if I finish the current PR (without all the "future goals") it still won't quite address the release management, but I hope to get to that next.

skaldarnar commented 4 years ago

Thanks for the effort and update on this @Cervator.

At this point I'm leaning towards either of gw or gooky as they are closer to our ecosystem. I still haven't found the time to look at gw in detail, but the use of PicoCLI seems promising. Most of all, it addresses my main pain point of the Groovy scripts being really hard to maintain or extend. I may use node-gooey as little playground to test out some features or compare implementations and performance in a PoC style, but will focus on bringing one of the JVM solutions to life.

Without looking at gw in detail it's hard to say which one to prefer. Kotlin seems to be the "modern" choice, being a bit more trendy than plain old Groovy. It also comes with some nice language features, is backed by Jetbrains, and Gradle seems to have a future with Kotlin, too. We'd need to find a convenient way of bootsrapping it, though. That's where gw can shine, just using the whatever Groovy ships with Gradle. On the other hand, it comes with some Java boilerplate as far as I can tell. Is it the safe but boring alternative?

Let me try to turn this text into pro/con lists for both gooky and gw:

gooky (Kotlin)

gw (Java/Gradle)

Cervator commented 4 years ago

Okay here's a quick stab at part 2: release management - but being that it is already midnight before a workday it is real brief (relative to my usual style)

I was able to spend a bit of time with TeraNUI, a brand new and pretty simple library of ours, thinking about and experimenting with the release management options. It mostly got stuck into an added https://github.com/MovingBlocks/TeraNUI/blob/master/gradle/common.gradle plus a pretty simple https://github.com/MovingBlocks/TeraNUI/blob/master/Jenkinsfile

Need to play with it some more and clean up (just realized the old isMaster method was left in), but in short it aims for simplicity yet makes fanciness available as needed. So it might qualify for the next phase approach for overall release management if applied to other things (with some more complexity - TeraNUI is easy in comparison)

In short: if version ends in -SNAPSHOT then publish will target a snapshot repo, and if not it'll target a release repo. No branch or Git checking of any sort. In the Jenkinsfile then there's a condition on the Publish stage on branch being master so CI builds of other branches and PRs won't cause published binaries. Whether it is ultimately master, develop, main or some other new name doesn't matter. It doesn't need the Git tagging approach to get involved either, but prepares for it (that'd be the next phase after)

You can trigger a release just doing the usual stripping of -SNAPSHOT from the version tag (in the root build.gradle in this case, bigger story there in other cases), commit, push to master. Bump to the next patch number, commit, push again to master and you'll get a new snapshot. If not setting to snapshot then successive pushes to master will cause build failures due to inability to re-publish the same release, as expected. But there is no more need for a funny module release management flag as there's no need for more than one branch or any needed differences between how modules handle their branches / releases differently (enabled or not)

I wouldn't go applying this to all the modules or anything yet though, not just from wanting to finish out the discussion, review, and so on, but also from wanting to see what GitHub does with default branch names. I'm curious to explore the Git tagging further, but that's sort of a different stage now.

More later along with some more POC stuff ... 💤

eviltak commented 4 years ago

My 2 cents for versioning: enforce git tagging somehow; makes it really easy to check out the exact source of a specific version of a module/lib. This isn't applicable to snapshots of course, but if we really wanted to find a way to correlate snapshot releases to their sources, we could somehow add the 7-digit SHA value of the commit corresponding to the release in the version number/package metadata.

skaldarnar commented 4 years ago

Update: Added reference to repo tool from Android.