Proposal for major reorganization of O3DE

Summary:

I believe a major reorganization of the engine is needed to take o3de to the next level.

I believe we should remove the engine object from the core o3de/o3de repo and leave only what we currently call the "Program Manager". The removed engine files should be placed in a separate repo o3de/engine. "Program Manager" application should be refactored and referred to only as "O3DE.exe", and for the rest of the RFC I will only refer to it as such. It is this application alone we should build and distribute as a "release". The customer then downloads this application via website download or platform package and runs it independent of the engine. Currently this application handles Gems quite well, however this application should be upgraded to treat all o3de objects the same as it handles gems, so there would be a "Projects" tab, "Templates" tab, etc. added so that it functions the same as the current gems tab, and yes even an "Engines" tab treating it the same as other objects. The user then uses this application to download the objects they wish.

The engine.json contains the information about this engine object including what dependencies it has. O3DE.exe responds to downloading an engine object the same as it would any other object, in this case the engine object has a list of gem dependencies, so it prompting/downloads this engines dependent objects. It places all of these objects, including the engine object to the default O3DE directories, i.e. ~/O3DE/Engines/o3de/23.04/ ~/O3DE/Gems/Achievements/1.5.3/ ~/O3DE/Projects/MultiplayerSample/23.05/ ~/O3DE/Projects/AutomatedTesting/23.05/ ~/O3DE/Templates/MyTemplate/1.0/ etc... for all O3DE objects. These of course are the default locations and can be overridden. O3DE.exe registers all these objects and you are off to the building races. The application gives the user the choice of downloading a zip release of an object (binary or code snapshot), if available, or cloning the repo at a tag, if available, etc.

This also means we should remove most, if not all, child O3DE objects from the core engine as the needed objects should be just set as dependencies for this engine object. These removed individual objects should be placed in their own separate canonical repos. Not every object needs it own repo, Atom, for example, is hierarchical collection of objects and would be best placed into a repo object with the same structure as the o3de-extras repo, so that all related atom objects can live in the same place. This way we can keep large systems of object together yet have them separate. They now live on their own timeline and DO NOT have to match the release schedule of O3DE. If there are no updates to the object but want to communicate compatibility, we may optionally alter the objects json to indicate compatibility/incompatibility with other objects as normal.

What is the motivation for this suggestion?

Making this change will allow the engine to use the object system to its fullest intended expression.

What are the advantages of the suggestion?

This will dramatically reduce the size of the engine and allow the user to only include what they need, which should also drastically reduce the engine download, reduce the footprint on the users machine and drastically reduce compilation times, promote best practices for the engine going forward. Make the engine more modular and easier to maintain. This should also dramatically change what we think of as a release as it just becomes building the O3DE.exe if needed, putting a tag on the engine repo and updating the engine.json. This will also dramatically reduce the deprecation process, as it will just mean removing a dependency line in the engine.json.

What are the disadvantages of the suggestion?

There will be more repos and we will necessarily have to change our processes to accommodate. Will we have to update a lot of documentation.

This proposal is a little light on specifics, so it's not easy to interpret without a lot of assumptions. It would be helpful if you provide these specifics to clarify the intension of this proposal.

What constitutes this Engine Object ?

It would be a little weird if the canonical o3de/o3de repo just contains the Project Manager, and not the rest of the engine.

The project manager depends on the engine and tooling code. How do you propose building is handled if you only want to keep the project manager in main repo? Consume the dependencies via submodule? Download prebuilt dependencies as part of build process of the Project manager?

Project manager could be the only thing distributed, but it would as you rightfully suggested, need a way to obtain versions of the engine.

There is already work ongoing to remove some less commonly-used gems from the o3de/o3de repo. This will already reduce build times for the main repo, as well as the size of the release package.

The engine is already very modular, so I don't see how moving things around as suggested will make it more modular. Even though a bunch of things live in the o3de/o3de repo, a lot of it are optional modules (gems). The modularity infrastructure is already in place. The necessary work would just be to move the optional things from the canonical repo, and provide a way for people who need them to easily fetch them.

I'm completely behind moving the optional gems and sandbox projects to a separate repo. However, I don't think anything else should be moved as that will introduce unnecessary friction for people building the engine from source.

I would be for having the core framework in o3de/o3de. But remember Project Manager != O3DE. You can use O3DE without Project Manager as O3DE is meant to support large scale CI/CD pipelines (typically command line) and the small studios (use Project Manager)

Still going thru this RFC, a lot more work would need to be done to lay the groundwork but I can see the impetus for suggesting something like this. From an actual technical executive perspective, I feel like moving the Project Manager to a new Repo and crowning that repo as the starting point for o3de rather than trying to pull or uproot the enormous amount of infra / jenkins / git / other stuff by transplanting the entire rest to a different repo to start with, supposing we wanted to go this path.

Going through this RFC, I don't think it should focus on trying to separate the Project Manager from the rest of the engine. The Project Manager is a GUI built using a combination of engine code and o3de python scripts to query o3de objects and and determine where they can downloaded from if remote.

From a technical perspective, it can work that the O3DE "core" can be made up of just the o3de python scripts and the CMake scripts needed to download the Python application in the first place.

Most of the engine is already modular with every Gem in the Gems folder being an optional piece.

I do think for organization purposes across repositories, the engine can be better served by organizing Gems into different feature verticals. Such as o3de-rendering-gems, o3de-scripting-gems, o3de-ui-gems, etc...

Also for re-organization is going to take place, the lack of modularity of the O3DE Code folder should be addressed in a way to make the engine lighter weight. I think the folder structure is good with respect for separating out features within the core engine code itself, however a lot more of it can be turned into a Gem. For example there is no reason Tools such as AssetBundler and AssetProcessor can't be part of a set of "o3de-core-asset-gems". Moreover the O3DE Editor would be better served if it was a Gem as well.

The very core of O3DE that would be part of the Engine, could be constrained to just be the necessary "Framework" static/shared libraries provide system functionality. Realistically the following two folders should be all the C++ source code that is needed as part of engine, https://github.com/o3de/o3de/tree/development/Code/Framework and https://github.com/o3de/o3de/tree/development/Code/Legacy .

Looking at other organizations like QT, they organize there repos based on feature modules and then provides a repo that combines the modules together for the purpose of building (https://github.com/qt/qt5). Perhaps that can be looked at as example

This also means we should remove most, if not all, child O3DE objects from the core engine as the needed objects should be just set as dependencies for this engine object. These removed individual objects should be placed in their own separate canonical repos.

I really like the idea of modular repos, but from a development perspective it's important to consider how separating frameworks, gems and testing projects into repos affects development. The more we break apart the repos, the more development and testing becomes difficult to keep in sync. For example, making a bug fix or change in a core library that requires changes to all Gems or all Projects becomes a lot more difficult. In some ways we experience this already with the content in o3de-extras lagging behind. If the "essential" gems, tools and projects we use for testing and bundling in the installer are moved out of the repository the core frameworks are in, be prepared for additional developer burden keeping them all in sync.

I think there could be a in-between step here where we start by moving as many optional/infrequently used Gems to o3de-extras and make them available as remote Gems. I started on this task with this draft PR, but it there are tests that need to be updated and other such details are needed to make this work.

o3de / rfcs