Open SrNetoChan opened 2 years ago
@SrNetoChan thanks!
Non-blocking issues that were not possible to fix in time can be added to a list of know bugs (Not sure if we already have that somewhere)
Just a quick note about that: maybe we might use the github tracker and tag the issues that affect a particular LTR release so that a list can easily and automatically generated.
@SrNetoChan thanks!
Non-blocking issues that were not possible to fix in time can be added to a list of know bugs (Not sure if we already have that somewhere)
Just a quick note about that: maybe we might use the github tracker and tag the issues that affect a particular LTR release so that a list can easily and automatically generated.
Sounds like a good idea to me. Maybe milestones are better? Not sure how they are being used now...
In general this sounds good to me!
My one objection is:
If possible, we will make use of the tester plugin to make testers' life a bit easier and speed up the testing. (The tester plugin can help prepare the stage for some manual test or specific verification)
I've some fundamental issues with how this plugin workflow has been designed, which I've described in https://github.com/qcooperative/qgis-core-tests/issues/3. Specifically I think:
For this reason I'd suggest we drop the tester plugin and and instead focus on just developing a comprehensive set of manual tests instead.
@nyalldawson I understand your concerns (we chatted about it once), but I believe we can reach an agreement :) .
Manual testing is a very exhaustive work, moreover in software like QGIS with so many use cases and multiple approaches for the same task. We will never be able to cover all the functionality with fully manual testing, we will lack enough manpower, so we tried to find strategies to overcome that.
Nevertheless, for some of the reasons you mentioned (python tests are "hard" to create and maintain), most of our tests will be completely manual, step-by-step descriptions to test one or more scenarios. So, let's consider the use of the tester plugin as complementary to those manual tests. For each test case (scenario), we should evaluate its usefulness and effort. The preferable way to create tests should always be the following:
CI Tests > Manual tests > Semi-automated tests
Maybe some of the examples we have created were not clear. My apologies for that. Here are some situations where using the tester plugin can be useful, IMHO:
Automated Integration tests. Sometimes tests could be fully automated and integrated in CI, but we need to use third party endpoints to be set, making it hard to do it on CI. We can use the manual descriptions to instruct the testers to prepare or configure the endpoints and then use the tester plugin to run the automated tests. Of course, that, if we discover that we can do it on CI, that's even better!!!
Semi-automated tests. When the steps to prepare the stage for the actual test case or visual verification are too long or too complicated to be repeated over and over in each test, especially when those steps have already been tested manually in another test case. (we can make this last part a rule)
I understand that this stage preparing steps could also be useful tests, but if we force the testers to repeat the same tasks over and over, we will be repeating ourselves and, in the end, limit the number of test cases we can cover.
It's also useful for organizations who want to run their own automated tests, specific to their workflows. Like running a set of models in a new QGIS version and making sure they work and return the expected results.
Moreover, test cases and test plans should be dynamic. Instead of having a final set of test cases we use forever, we should keep evaluating the usefulness of each test case, if it fits better in CI, manual tests or semi-automated tests.
We should also evaluate adding or removing test cases for a specific release. Let's say that in a particular release we know we did some complex changes to a part of our code or a completely new feature, and we want extra testing, we should create more test cases for that particular feature or workflow. Or if we know we have bumped a dependency, let's try to make more tests that rely on it. Creating new test cases will be open to everyone, to include them in the test plans for a particular release must be a balanced decision, and we, "testing team", will definitely need the help of developers and package manager to better decide.
@nyalldawsonI agree in principle but manual tests are often complex workflows with many preparation steps and this is the case where the tester plugin can become handy because it can automate the preparation steps and reduce the manual steps and finally the total testing execution time.
Another difference between CI tests and what the tester plugin can do is that the CI tests are mostly unit tests with a mocked QGIS app while the tests we are talking about here are run in the real QGIS desktop application.
QGIS Enhancement: Manual Testing on LTR Releases and Release Schedule
Date 2021/12/21
Author Alexandre Neto (@SrNetoChan)
Contact alex.f.neto@gmail.com
maintainer @SrNetoChan
Version QGIS 3.22 +
Summary
By a PSC decision, it was decided that we as a project should go forward with the work started on #180. This is a summary of what has been settled in a meeting with the following participants:
Release plan and messaging
Testing period
Test plans
Tester plugin
If possible, we will make use of the tester plugin in the following cases:
Further Considerations/Improvements
Not mentioned in the meeting (subject to approval)
Votes