Closed allisaurus closed 1 year ago
lmbr-pip
commented
2 years ago SIG/Network should probably request a new HttpRequestor like Gem as the current one is provided as convenience and only works on some platforms (as well as for the reasons highlighted above).
ValorZard
commented
2 years ago From my brief experience with this engine, and by talking with people about this engine, one of the main factors I see in it not picking up this engine is the idea that this sis till an Amazon product. They often point to code in the base that isn’t open source and is owned by Amazon. If the Amazon gems were removed, and the entire engine codebase was all licensed under Apache 2.0, it would make using this engine much more appealing in my opinion.
lmbr-pip
commented
2 years ago SIG/Network questions:
allisaurus
commented
2 years ago Does Twitch or other gems have other third party libraries to migrate?
The AWSNativeSDK appears to be the only 3rd party library in use by the Twitch, HttpRequestor, AWSClientAuth, and AWSMetrics gems.
The AWSGameLift gem further uses the AWSGameLiftServerSDK, which will need to be migrated as well (since it's the only gem that uses it).
AWSCore uses a number of other 3rd party libraries, but they're all in use by other gems too (Qt::Core, pyside2, etc.)
byrcolin
commented
2 years ago The objects we remove can either be their own repos or we can combine them in one repo. To fight sprawl we should combine them into one AWS repo. I have instructions on how to accomplish this without losing history.
Example moving a gem object from the engine to some other repo, in this case a aws o3de repo with history: For the AWS repo we want a structure like this:
root
\
Gems
\
AWSClientAuth
AWSMetrics
...
\
Projects
\
AWSProjectGoesHere
...
\
Templates
\
AWSTemplateGoesHere
...We can do them all at one time or just one object at a time. Lets consider doing multiple at a time using Gems\AWSClientAuth and Gems\AWSCore as examples:
We need a clean clone of o3de/o3de. Since we are doing multiple objects at once I would copy this newly cloned repo as many times as objects I had to move.
git clone https://github.com/o3de/o3de.git
copy o3de o3de-awsclientauth
copy o3de o3de-awscore
... We don't need to move Gems\AWSClientAuth or Gems\AWSCore into a different subdir as they are already where we want them to relative to the root in the destination repo. In the Gems\AWSClientAuth , but if it wasn't we would add --to-subdirectory-filter Gems/ or were it should go...
We filter out the rest of the repo:
cd o3de-awsclientauth
git filter-repo --path Gems\AWSClientAuth
cd ..\o3de-awscore
git filter-repo --path Gems\AWSCore
... Now o3de-awsclientauth looks like this:
root
\
Gems
\
AWSClientAuth And o3de-awscore looks like this:
root
\
Gems
\
AWSCore Now we need a clean aws lumberyard o3de destination repo (I'm not sure we have a name for it yet), and for clarity ill rename it aws-extras
git clone https://github.com/<whatever it is>.git aws-extras
I'll add the filtered repos as remotes to our o3de-extras
cd aws-extras
git remote add awsclientauth ../o3de-awsclientauth
git remote add awscore ../o3de-awscore
...Update the remote:
git remote update
Create a new branch so we can PR this back in
git switch -C inital-aws
Merge allowing unrelated histories
git merge --allow-unrelated-histories awsclientauth/main
git merge --allow-unrelated-histories awscore/main
...Push
git push origin inital-aws
You then have to take care of any lfs objects by uploading them.
Remove the remotes:
git remote remove awsclientauth
git remote remove awscore
...Then create your PR. Once accepted you then delete Gems\AWSClientAuth and Gems\AWSCore from the o3de/o3de repo or just remove the references in the engine.json to AWSClientAuth and AWSCore for the time being. Create your PR for its removal and done.
As for AR in aws repos, there will be no o3de AR on any non-o3de repo. O3DE AR only occurs on o3de/o3de and soon o3de/o3de-extras and possibly other o3de repos in the future but not yet. Any non o3de repo AR is the responsibility of the external partner to test.
I do not believe it is necessary to change the name of the gems at all. We can keep them in there and simply remove the references to them from the engine.json and the engine will no longer know about them and we can keep them there for a period of time if we wish, I would remove them so there is less confusion. We can simultaneously add the new AWS repo. If we want to keep the objects for a while, we should investigate if we can put a lock on adding and changes to those folder for a period of time.
We will need to do a pass on all o3de projects and make sure the AWS gem references are removed if they are not needed.
The 3rdparty libs can stay where they are at the moment in the o3de, we can move them to an aws server/account and add that server to the LY_PACKAGE_SERVER_URLS later.
The new aws lumberyard repo entry will be added to the o3de engine.json and so will be discoverable by the program manager/cli. I disagree with the characterization that o3de will lack "out of the box" cloud integration or that discoverability will be reduced. It is our job to make the discovery and downloading of the external gems a first class transparent an simple experience as possible. Since they are canonical objects we can shortcut and automate downloading and registration. Further this will also benefit non Amazon cloud providers o3de partners which will now have an even playing field with aws's offerings.
The docs for aws objects are aws's concern and should not live in o3de at all.
allisaurus
commented
2 years ago @byrcolin thanks for weighing in here and for the run-down of how to retain the commit history post-extraction! I agree it should remain intact before/after this effort.
To fight sprawl we should combine them into one AWS repo.
I'm not sure I agree, or at least I'm inclined to limit the scope of what we group together. It would buy us some testing efficiencies to house AWSCore and the gems that rely on it in the same repo, since they could then share a testing project and AR resources (like how AutomatedTesting works now for o3de/o3de). But it would also mean that development of the individual gems would be coupled and potentially unrelated changes (like a metrics pipeline change) may block others (like a game lift feature). But maybe this won't be a big enough problem to outweigh the benefits, especially if game/simulation projects can still consume the gems individually, w/o cloning and pulling down the whole repo.
As for AR in aws repos, there will be no o3de AR on any non-o3de repo. ...Any non o3de repo AR is the responsibility of the external partner to test.
100% agree, the destination repos should have their own AR and release processes. Removing the gems from O3DF's AR is listed above as step 3 under "Feature design description".
I do not believe it is necessary to change the name of the gems at all. We can keep them in there and simply remove the references to them from the engine.json and the engine will no longer know about them and we can keep them there for a period of time if we wish, I would remove them so there is less confusion.
How does migration from one version of the gem(s) to the other work then? I expect the order of operations would work like this, but maybe I'm wrong:
Can step 4 can be done first (or simultaneous with step 1) w/o running into problems? I think maybe it'll depend on the project code, but maybe not if the gems' public APIs are exactly the same.
The new aws lumberyard repo entry will be added to the o3de engine.json and so will be discoverable by the program manager/cli. I disagree with the characterization that o3de will lack "out of the box" cloud integration or that discoverability will be reduced. It is our job to make the discovery and downloading of the external gems a first class transparent an simple experience as possible.
This is the part of the equation I had the least insight into when authoring this. We should definitely leverage whatever discovery mechanisms the community supports once these gems live external to https://github.com/o3de/o3de
lmbr-pip
commented
1 year ago Accepted in SIG-Network meeting July 19th. @lmbr-pip to archive.
Summary:
Currently there are 6 Amazon-specific gems within the main o3de/o3de repo: AWSCore, AWSClientAuth, AWSMetrics, AWSGameLift, the Twitch gem, and HttpRequestor (which is essentially a wrapper around AWSNativeSDK types and methods for making REST calls). These gems are unique in that they are optional and vendor-specific, yet live within the main engine source code. This RFC proposes extracting these gems to their own, Amazon-owned GitHub repositories and removing (over time) the versions of these gems that live within the engine source. This will improve the vendor neutrality of the engine itself, while allowing for development of these gems to continue in a leaner, more decoupled way from that of the entire engine.
What is the relevance of this feature?
This work will change where 6 gems within o3de/o3de are housed, how they’re pulled into O3DE projects, and (potentially) how they’re advertised to the community. The biggest impact, from a user perspective, will be that once this work is completed, users will need to register the individual Amazon gem repositories with the project manager before being able to see and select them for inclusion in a project (unless other mechanisms are identified - see “open questions”, below). Users of the existing gems would need to migrate to use the external gems in order to continue pulling in the latest fixes & features.
Feature design description:
At a high level, this work would entail making the following changes, in order:
Technical design description:
The new “external” Amazon gems should be functionally equivalent to the existing source versions of the gems, but will need to differ slightly in terms of build setup and AR due to their new locations. Those differences may include:
gem_namevalues and CMAKE target names, so that they don’t conflict with the source versions of the gems during the initial period of time when they may both exist in the same project.LY_PACKAGE_SERVER_URLSvariable.Adding the external versions of the gems to an O3DE project will be done as it is now for external gems. See the relevant documentation pages:
What are the advantages of the feature?
Advantages for O3DE:
Advantages for Amazon Gem users:
What are the disadvantages of the feature?
Are there any alternatives to this feature?
An alternative would be to leave the Amazon gems in the main o3de/o3de repo and continue to maintain them there. However, this may set a confusing precedent for adding similar optional vendor gems in the main engine source, which over time would encumber maintenance and testing of the core engine. For example: asking SIG-Network maintainers to incur costs on personal AWS accounts, or other commercial platforms, to fix bugs in the main engine repo is not fair or sustainable.
How will users learn this feature?
Advanced notice of the Amazon gems’ removal from the main engine repo and comprehensive migration instructions will be available and circulated at least one release cycle prior to its execution. As mostly enumerated above in “Feature design”, we plan make the community aware of this change by:
Are there any open questions?