[Discussion] Quality Assurance Discipline

[brainsick]

Unit testing and quality assurance are two very different things. I might summarize the most important differences as:

• A QA test is written based on the behavior defined in the documentation. Any behavior that doesn't agree with the documentation is a bug. Said bug could be in the code, the documentation, or even the test itself; some triage may be necessary.
• A QA test can operate at any level of the stack. Everything from the behavior of the API to the behavior of the GUI is fair game.
• A QA test can be written in response to an issue.
• A QA member would report bugs in the issue tracker. If they can also fix it, great! This shouldn't be expected. Bug reporting can sometimes be automated.
• QA test suites can be automated but this is often done outside of the continuous integration system.
• QA can be coordinated with Release Engineering. The QA team would run their test suites and "authorize" a release.
• A QA discipline adds process. Coordination between core developers/release engineering/quality assurance is important.

I've put together #3 as an example of a very basic QA test. It can be run from the command line or integrated into a future test harness. If you'd like to help shape what a QA test might look like in Godot, please chime in on the PR. I'm cataloging implementation issues there.

[NathanWarden]

I like it and agree that QA and unit testing are different. However, while different things they are still very related.

I've heard it this way: The QA engineer does the actual testing and the unit tests are actually just checking.

I agree with all points except I would pretty strongly disagree with the following statement at least until there's an actual stable QA team in place for Godot but probably even after that as well:

QA test suites can be automated but this is often done outside of the continuous integration system.

This is one of the main reasons why CI has been so heavily adopted. It gives immediate feedback to both contributors and maintainers whether a particular commit ought to be accepted and allows it to be fixed before it ever reaches the master branch. If these tests are run arbitrarily by a QA engineer, or every once in a while in an automated system, it means that the regression has already made its way into the master branch and has already affected somebody. For instance, if I remember correctly from reading the Unity blog that they use the CI system to catch regressions before they even reach QA, which gives the responsibility back to the developer to fix their bug and not bother the QA team, unless there's a dispute as to whether the test itself was written incorrectly (since sometimes they are).

Overall, I believe using the CI system to run Unit Tests will save everybody a lot of work in the long run.

There is no real QA team in place for Godot yet. I know QA engineers are far more important than an automated system as they tend to find actual bugs where an automated system only can check against regressions, but an automated system is far better than nothing in place for an every once in a while QA team.

[NathanWarden]

Let me amend my previous statement by saying that certain types of automated tests can take a very long time to execute since they may involve opening large projects or some other complex thing. Those should definitely be run outside of the CI system on maybe a daily basis.

[brainsick]

@NathanWarden I'm trying to be a little conservative and I'm kind of floating between different terms/concepts in the initial post. My goal was to simply establish that unit tests are different from QA tests.

There are lots of ways to incorporate a quality assurance discipline. That discipline can and should evolve over time as it matures. Right now, we're on the fringe making requests of the core team. The core team has humored us and set up a repository. I think that's about all we can expect at this time. It's up to us to establish the value of a QA discipline.

To that end, I'm trying to collect data, catalog issues, and present them for further discussion. In the attached pull request, I've written 26 assertions and identified 3 bugs. I think that helps establish value.

I'm not a QA guy myself although I've interacted with them plenty in my career. I'm putting together what I imagine a QA test/suite might look like. I could be off the mark.

[NathanWarden]

I'm not a QA guy myself although I've interacted with them plenty in my career. I'm putting together what I imagine a QA test/suite might look like. I could be off the mark.

@brainsick Thanks for getting that started :) It definitely needs to start somewhere. In the longterm I believe we can actually make Godot more stable in the parts that have automated testing in the master branch more stable than each previous stable release even when compiling HEAD from source. Obviously the new feature portions will almost always be buggy.

I'm not a QA guy myself as a profession either, although I have implemented QA principles and systems into many of my projects. I can also say that unless the QA system, especially automated test suite, isn't run on a constant basis it will have failures in many places to the point where nobody will want to fix it. At that point it's easier to scrap everything that's broken in the test suite and start over in those places... or people just start ignoring the test suite. If the test suite is incorporated into the CI system you completely remove this problem altogether because each contributor must make their code succeed in the test suite. If the test itself needs to be changed, the contributor should be skilled enough to update the test since it will be written in something higher level than C++.

With that said, I've never incorporated CI into my workflow as I haven't found a need for it as they were all smaller projects, but I need to for a larger project I've been working on for a few years now with some friends of mine. If I figure that out and running this automated test suite hasn't been incorporated by then, I might take a stab at it for Godot... no promises though :)

Thanks again so much for getting this started as this is something Godot will seriously need if it's to be known as a stable engine. :)

[wtarr]

Just want to tag on some of my own ramblings, as this is an interesting topic. Feel free to agree/disagree, discussion is good.

Theoretically, every bit of code being committed should have a covering unit test, it at least shows what you have committed actually works i.e. you are returning the expected value for a known set of inputs. Practically, this soon can become a nuisance which people tend to avoid as it feels like writing code twice, I know this well from my own profession. But in the long run the benefits out weigh the negatives as I have caught many a regression of my own that could potentially have made it into a code base.

I work with C# as my day job and I find the tooling is there and easy to use (i.e. nunit and its test runner). This of course takes away a lot of the pain for developers running the tests before committing, we also have CI in the background from the likes of Jetbrains teamcity. So I think godot will need some sort of test runner / tooling also to make the process of running the tests that bit easier for developers, speaking of course if gdscript is the choice for writing tests, I cant really speak for c++. I guess a standardized format for the tests should also be considered/established.

I think also having unit tests for godot would be a nice way of introducing newcomers to the code base, I know myself because I am not all familiar with c++, so making code changes for me would be daunting. I also think it would act as a nice way of self documenting the code.

Just noted from above, where there is a mention of tests that can take a long time, I think really these are no longer considered unit tests and should possibly be treated as integration tests which would probably require a separate process. Maybe unit tests could be run on every commit and integration run once a day?

[juan-garcia-m]

The difference between Unit Testing and QA is the visibility.

Running unit tests ideally should be part of of the build itself. The tests can have privilege access to some internal functionality that is not meant to be exposed publicly in the API. This is white box testing.

For QA, the testing is done over the exposed functionality. Automating is more complicated as the access scope is reduced to that of the end user in most cases. This is black box testing.

Assuming that this repo is intended as QA and not Unit Testing, this is a way to proceed:

Automated Tests

Create or adopt a test framework

For example, here is one https://github.com/bitwes/Gut Having a test framework will split the complexity. Adopting or creating a test framework that can be reused between Godot QA and developers games unit testing would be useful.

Create a test for each class in the documentation

This can be: tests/api

Each class can have each own directory, for example tests/api/Sprite .Each function should have as many tests as needed. This can grow when additional issues get reported in the main repo and no test was made for that issue previously. Tests must be updated when API changes, as it should do the documentation. Those kind of tests are useful to keep API documentation up to date.

Create additional tests for common use case

This can be: test/misc

Some issues can only be reproduced when combining several different classes. Special test cases can be created to cover that.

Create performance tests

This can be test/perf

While all the tests should be timed, performance tests are there to measure how specific behavior has been optimized. This can help developers to avoid early or blind optimizations.

Batch execution and reporting

All the tests must be discovered by a main script and run in batch. The main script should expose an API to the individual test cases, which will collect info about the tests. At the end an automated report has to be generated including test passed, failed, skipped as well as the time each test used.

Non automated tests

This should be a check list which each item is a series of step to reproduce. Those are tests that are not easy to automate. For example issues with the shader results, garbled images, etc. This kind of test can also be step by step checking the editor GUI for different kind of tasks. Assets can be provided for those kind of tests.

---

Those are just some ideas for how to get started. I do have actual QA experience, so maybe I can help somehow.

[Xrayez]

2 years later bump: we've integrated doctest framework in Godot now in 4.0.

• A QA test can be written in response to an issue.

I guess godotengine/godot#40795 could qualify as an example for this too?