Roadmap for moving Iota modules to the engine repo

[skaldarnar]

Motivation

Terasology has a lot of modules, and working on bigger refactorings or large restructurings is cumbersome when changes need to be split over many repositories. (many PRs, many reviews, merging in the correct order, ...).

The goal is to achieve better development efficiency and higher velocity.

Proposal

We move the modules of the Iota line-up into the engine repository.

• BiomesAPI
• CoreAdvancedAssets
• CoreAssets
• CoreRendering
• CoreSampleGameplay
• CoreWorlds
• Drops
• Explosives
• Furnishings
• Health
• Inventory
• ModuleTestingEnvironment

Concerns

• we make some of the modules more "special" than others, potentially further complicating our build setup
• we don't have a clear target picture for the CI setup yet, so whatever we build here may be helpful or hindering
• we may loose the history of changes in those modules
• in case we can keep the history, commit messages might be a broken (for instance, a commit message like doc: add component and ui element docs (#102) from Health would refer to #102 instead of Terasology/Health#102.
• the module line-up might be too fine-grained as is, and we might change it again rather soon
• the modules will no longer be visible on the website
• the module docs will not longer be served via github pages (for example, https://terasology.github.io/Health/#/)
• as the engine repo is in a different org, we cannot transfer tickets.
• we loose open PRs on the modules
• unknown complications due to used namespace and/or wrong versions being picked up?

Task Breakdown

• [ ] Merge the module repos into the engine repo to modules/*.

  See How to Merge Two Git Repositories (stackdiary.com) and How do you merge two Git repositories? for preserving their git history.

• [ ] Archive the Iota module repositories.

  The modules should be moved to https://github.com/Terasology-Archived. Ensure that we are not losing any crucial information in tickets or wikis. Since the engine repo is in a different org, we would have to create new issues and copy over the content / summarize any discussion.

• [ ] Ensure that game bundle builds are still working.

  The engine package is build via gradlew distForLauncher (https://github.com/MovingBlocks/Terasology/blob/e33518e3a04a901de4796aa0cf68c8614bcc26cb/Jenkinsfile#L54C82-L54C97) which is archived as an artifact and then used as base when assembling the game distribution package. Additional modules (from separate module repos) are added on top of this base.

  To include the modules from the engine repo in the final distribution package we need to make them somehow available to the job bundling the final package. Some options I see are

  • include the modules in the Terasology.zip produces by gradlew distForLauncher such that they are automatically picked up when using this ZIP as a base for the final bundle
  • add the engine modules to the archived artifacts, and unpack them in the iota and omega jobs explicitly (hard-coded)
  • publish the engine modules such that they can be pulled in the same way as other modules via Gradle dependencies in the distro job

Additional Notes

• Do we want to rewrite the repo histories to fix references (for instance, PR referenced by number like #102)? This would mean quite a bit of additional effort, and I'm not sure how hard it would be to get this re-write right...

[PurityLake]

I'm wondering if we could do a squash merge commit for each module with a link to a document pointing to issues and PRs for the respective repos?

I feel it would be very possible with a script to parse the git repo of each module to generate a markdown document.

edit

Actually one thing a pygame-ce did once the fork was created, was to "copy" all completed issues and PRs (in progress PRs could not be done), linking up the issues and PRs by changing the numbers they pointed to, to the new ones. This may be less desirable though as this is not a fork with no outstanding issues or PRs.

[skaldarnar]

We discussed this roadmap in more detail with @BenjaminAmos , @jdrueckert , and @skaldarnar on August 13 on Discord. In particular, we discussed the mono repo idea, motivation and impact in some more detail and came to the conclusion to not change anything about the current setup, including not to move modules into the engine repo and not to move a bunch (or all) of them into a new shared repo.

We asked ourselves what the main driver for moving modules to the engine repo was/is, and ended up with developer efficiency as main argument. We are in particular concerned about large scale changes and refactoring (like JOML), and the overhead the multi-repo setup brings with that.

I think this was in the context of possibly changing the module API, as in the past (or in the case of NUI, in my personal experience) it's been quite onerous making those changes individually. With good tooling for git though it might not be as necessary.

However, we believe that we can address large scale refactorings by relaxed review processes, better git tooling (git and gh), and potential custom tooling (groovw scripts).

If the changes were mainly automated though, with a low risk of breaking things (like renaming packages), I think we could just agree to push direct and hoping that it still compiles in such cases.

In that context, we also discussed the workspace repo idea again. I will create a separate roadmap ticket for that to share the details.

I'm closing this issue based on the results of the discussion as summarized above.

[skaldarnar]

@PurityLake welcome, and thanks for your feedback!

I'm wondering if we could do a squash merge commit for each module with a link to a document pointing to issues and PRs for the respective repos?

That's a nice idea, the document could be the archived repo (I guess that's as stable as anything else), or some document we commit in the module itself (similar to the good ol' CHANGELOG files).

[…] was to "copy" all completed issues and PRs (in progress PRs could not be done), linking up the issues and PRs by changing the numbers they pointed to, to the new ones.

Agreed, and it might have been applicable here as we are merging repos. All this does not sound too complicated, but still would have needed someone to put the time into it. With the decision to leave it as is for now (see https://github.com/MovingBlocks/Terasology/issues/5129#issuecomment-1685304479) I would rather not invest into that right now (but keep those good suggestions coming on all the other issuse we're facing 🤓 ).