RFC: Re-factor the pytest.Class objects in the editor_test framework code.

[jromnoa]

Summary:

• A single pytest.Class object that shared/batched/parallel/single test classes can all inherit from and share.
• This RFC was spawned as a byproduct of the work in https://github.com/o3de/sig-testing/issues/27 as the pytest.Class object proved far too difficult to convert over, requiring an RFC of its own.

What is the relevance of this feature?

• Why is this important? This is important to help limit tech debt and ease the burden of long term maintenance for the test tools. As it stands now, most of the code that both the Editor and upcoming (https://github.com/o3de/o3de/pull/9609) that is utilized for the pytest.Class portions are copypasta with the editor fixtures swapped out for material_editor fixtures instead on the ly_test_tools side.
• What are the use cases? The use cases are for any new O3DE executables that we want to test using these batched/parallel test tools. For instance, if a team (let's say Atom) adds customizations to their code in the new MaterialEditor files that have the potential to benefit all shared tests but that code is on the pytest.Class object, then other test classes with their unique pytest.Class objects will not inherit these benefits (such as the Editor). We'd have to manually add the code to each object (MaterialEditorTestClass & EditorTestClass). Additionally, when we want to add a new feature it will be much easier than it is now since parsing through the nested for loops and callback functions within those for loops gets quite tedious (it mixes some fixtures in there as well to get data/values from further complicating things as far as re-factoring or adding new functionality).
• What will it do once completed? Have a singular pytest.Class class that new classes inherit from will ease the burden of adoption as well as ease fixing future bugs or functionality within the code.

Feature design description:

• The Design: I haven't had the chance yet to fully lay out the architectural discoveries I made with the editor_test framework when working on the other RFC, however this new class would ideally go into the newly created multi_test_framework.py file, similar to the other objects & functions that were moved and created. You can see a full list of changes in the PR for that work here: https://github.com/o3de/o3de/pull/9609
• After the new class is added to that file, the editor_test.py and material_editor_test.py modules will inherit from it for their specific EditorTestClass and MaterialEditorTestclass objects respectively. Not everything will be able to be flattened and inherited by both, but anything that can be should be moved over. Ideally some __init__() attribute setting, pytest fixture parametrization, or similar implementation can be utilized in the editor_test.py and material_editor_test.py modules for setting specific values for Editor or MaterialEditor.
• Some pitfalls to watch out for discovered during the previous RFC work on MaterialEditor: Fixtures primarily. I tried to get the fixture calls in the correct order when converting the pytest.Class object but it become non-trivial when swapping between MaterialEditor and Editor tests. Make sure, at all phases of the re-factor, that you are actively running and checking the test code as you iterate. I failed to do this on my first attempt at the MaterialEditor RFC and it buried me in a messy callstack. Update the code, then run the code (mainly because of the fixtures) one step at a time.
• For additional use cases, simply run the existing tests we have in https://github.com/o3de/o3de/tree/development/AutomatedTesting/Gem/PythonTests and make sure they work the same as they do now.

Technical design description:

1. Write out a new architectural chart for how the existing objects function and where we want our new pytest.Class object to go to modify this existing architecture. This will help keep the work focused and guided before diving deep into the coding aspects (perhaps a future RFC review can contain this effort, for all of @o3de/sig-testing to review together).
2. Create a new class in the new multi_test_framework.py module. Name it something similar to the other class objects in there (i.e. AbstractTestClass which we'll go with for the rest of this design description to make it easier to follow).
3. Have the AbstractTestClass inherit from pytest.Class similar to how EditorTestClass inside editor_test.py does: AbstractTestClass(pytest.Class).
4. Add all shareable functions and code into the new AbstractTestClass object.
5. MaterialEditorTestClass and EditorTestClass should inherit from AbstractTestClass: MaterialEditorTestClass(AbstractTestClass) & EditorTestClass(AbstractTestClass).
6. You will probably need to do a super().__init__() call for these classes after inheriting from AbstractTestClass.
7. Do not break any existing interfaces and do not rename anything that is used by the tests inside https://github.com/o3de/o3de/tree/development/AutomatedTesting/Gem/PythonTests since we want to make adoption of this for test writers as least disruptive as possible (emergent work kills sprint productivity).

What are the advantages of the feature?

• Ease of creation of new features as well as maintenance or bug fixing for the editor_test framework code.
• Ability to easily add any executable as a new potential test target with our test tools.
• Implemented as part of ly_test_tools so we get to reap all of the benefits of the existing test functions and code by simply expanding upon it with these new objects.

What are the disadvantages of the feature?

• Complication of the re-factor work required here means that the architectural planning for this needs to be accurate before the work is taken on. Make sure to chart out the objects in the existing framework code and ensure they are understood then share them with the team and how this new re-factor will play into that going forward.

How will this be implemented or integrated into the O3DE environment?

• It will be a simple PR to update the existing code, then it will be automatically adopted behind the scenes (backend tech debt change).

Are there any alternatives to this feature?

• An alternative in the meantime is to simply do a copypasta + tweak of the existing pytest.Class objects for new executables that need to utilize these framework tools. While it's not ideal and won't help with long term tech debt woes, it can solve the problem of having no functionality at all by taking this approach - now is better than never: "Now is better than never. Although never is often better than right now".

How will users learn this feature?

• They can review the new framework design documentation that will go out and add new features or fix issues they see with the existing objects and functionality we have in place.
• This will hopefully deepen existing users' understanding on the batched/parallel test framework code which is a net benefit for all users of these tools.

Are there any open questions?

• Is this important to do right now or since it's tech debt do we wait before adding it?
• How do users feel about existing documentation?
• Should this integrate more seamlessly into the existing ly_test_tools or should it remain kind of inside its own section like it is now in ly_test_tools?

[jromnoa]

I found a way to work around this actually so it won't be needed. I was able to split this class into its own object and successfully launch with it.