Molecule: Generalized versioning, change tracking, build & update system

[Frooxius]

Is your feature request related to a problem? Please describe.

There is currently a number of problems on Resonite that share the same common set of problems:

• Resonite builds are done and pushed manually, as well as compiling of change notes and contribution credits. This also limits ability to test, switch different branches/builds, revert to older versions or be on more stable branches for companies
• The distribution & update methods for in-game content are limited to shared folders. This makes versioning & updating of these items/worlds difficult and laborious
• Once ProtoFlux nested nodes & libraries are introduced, there will be need to resolve dependencies for these (e.g. when two or more ProtoFlux systems depend on the same ProtoFlux library)
• Plugins/mods extensions face similar problem, requiring manual install & updates
• Once Prefab system is introduced, it will need a way to update the instances of the Prefab and determine when update is okay to happen (e.g. when non-breaking changes happen) and when not (e.g. when new major version of the prefab is released)

Describe the solution you'd like

We have plans for a generalized system for handling versioning, change tracking and updating, dubbed "Molecule". This system will solve these problems in a general way, with specializations for different usecases.

The system has several concepts:

• Snapshots - these are roughly equivalent to commits - they represent a particular "version" of given build, library, item, prefab, module... Each snapshot has a list of changes done in that snapshot compared to the previous one
• Builds - these are "installable/spawnable" versions of given snapshot. Given snapshot can produce multiple ones - e.g. for Resonite snapshot, there's Windows graphical client, Linux graphical, Windows headless, Linux headless and so on... For in-game items, there can be the "edit" version as well one that's been auto-optimized for different uses
• Branch - snapshot belong to branches, which are roughly equivalent to Git branches. This allows to separate out development versions, that can exist in parallel, e.g. when large features or changes are being done over time
• Releases - these are specifically tagged Snapshots. Each release has a "channel" associated to it. If user is subscribed to a channel (say the main one), they will always get the latest release in that channel - allowing the system to auto-update

With these concepts, the system can be used and queried in a number of ways, with specific use-cases built on top.

• For updating Resonite user can install a number of instances. Each instance can be subscribed to particular Release channel or install a specific Snapshot
• For Items/Prefabs/Libraries/Modules, the in-game instances can be configured to only update if minor version changes, but not when major one does. If creators follow semantic versioning, this should allow for automatically fetching the latest compatible version of prefab/protoflux-library/webassembly module
• Whenever a given channel updates, the system can iterate over all snapshots between the last one and new one and generate a full changelog - this is not just for Resonite builds, but also whenever content creators update in-game items/worlds/libraries and so on
• Whenever we release updates, automated jobs can be ran with that release - e.g. for Resonite builds, they'll be automatically submitted to Steam, Itch.io, Linux packages, changelog posted to Discord and other socials, GitHub issues closed and so on

This system will also integrate with other tooling - e.g. snapshots for Resonite itself and its libraries will be generated from conventional commits. This will simplify writing changelog and contributor references and credits for reporters - they will be auto-generated and formatted for different media (Discord, Steam, Reddit...).

Describe alternatives you've considered

We've considered utilizing some existing tooling and update systems, but these would not provide the same generalized level of support and integration that our own system would be able to handle.

Additional Context

This is pretty important system to prioritize on our end, as it will allow to save a lot of repetitive work with producing builds and updates.

It would also resolve certain problems with updates (e.g. https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/1139) - those would no longer be done manually, but fed into the system and processed and tested before they are merged in.

[charlie-sans]

hmm, if i made a script to generate patchnotes from git diffs, would you check it out? i've been meaning to do this for a bit now but it could be handy?

[charlie-sans]

what about dynamic module loading? being able to update stuff live so people don't have to restart stuff for minor changes?

ether I'm reading it wrong or just blind lol

[Frooxius]

@charlie-sans This would help you fetch new modules easily, but it doesn't do anything with dynamic loading - that's a completely separate issue. The responsibility of this system is purely to distribute updated versions, what is done with those is external.

[Frooxius]

I'm also noting another use-case. Currently we are using hardcoded inventory locations for templates for certain items - default video players, audio players, color dialogs and so on.

This makes testing and updating more difficult, because we can only replace one copy globally and we cannot do gradual rollout or testing on beta branches.

Once this system is in place, we will be able to utilize it for the default item templates as well and utilize its versioning ability to ensure that the correct version can get fetched.

[charlie-sans]

nice! ive got some more ideas but i can't remember them. ill let you know if i remember!

[JackTheFoxOtter]

As it was brought up in #1619, I'm curious what the scope of Molecule is in regards to change tracking / merging. Would the use-case of having, to take an example from that ticket, a facet maintained by the content team, forking it into a private branch and modifying it, then once a new version of the base branch is released by the content team comparing changes and potentially merging them into the personal fork / potentially resolving merge conflics be covered by this?

[Frooxius]

Change tracking is one of the primary functions of Molecule. But merging isn't its responsibility - that is something that will have to be built on top of it.

Meaning there will need to be a system that when Molecule says "There's a new version of XYZ", it is able to update that version and re-apply the customizations. This would be handled by the prefab system.

[shininghero]

Will this also allow queuing updates for release on specific days? A 'Patch Tuesday' release schedule would be beneficial in the long run, especially as the platform gets more popular and starts drawing in big events.

[Stellanora64]

@shininghero he said "It would also resolve certain problems with updates (e.g. https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/1139)" which is an issue related to what you are describing. So from my understanding yes.

[Frooxius]

@shininghero It'll make it easier to switch to a process like that in a few ways:

• Any changes to the code will result in testable builds on separate branches, without additional effort on our end. This means that we can jump on these earlier than usual and start testing impact of the features before it gets pushed out
• We can develop features more in parallel too, so if any changes are causing issues, they get held up before they get merged. Whatever ends up getting QC approval gets in the next build
• We can have more long term support branches, that update less often for specific purposes
• Once changes are done and builds ready, we can easily hit a button to switch the main branch to whatever the latest tested build is, without being blocked on further development and changes on other stuff (right now, this is a bit of a problem and one of the reasons stuff gets pushed out as soon as it's done and ready)

However there's one aspect that's more social, that's a bit problematic, given the size of the community we have, that might still be harder to resolve.

If new features are ready to use and potentially already available in the testing builds - will people want to wait a while before they can use them on the main branch? Say we get something done on Tuesday, but it doesn't make it in - you now have to wait a week for that fix or new feature to get in the main build.

Alternatively you can use the "prerelease" branch, but then you're fragmented from the rest of the community. With the community being small, this can become a problem.

For that, I'm not sure if we'll specifically do something like patch Tuesday or something like that, because it trades off some of the agility for things. The structure we find might still have more frequent updates. But the main thing, this tooling will give us the flexibility to explore those.