petersilva
commented
5 years ago background and current testing strategy:
-
developers make changes in issue branches, and never commit directly to master.
-
the developers run the flow_test, which includes all available unit test, as well as end to end integration testing of a reproducible network of all existing components in a variety of configurations. Perfect result on the flow test (all tests passed) is a condition for acceptance to master branch. The developer should pass the test on one laptop, and a second person on a different machine reproduces the flow_test results as a pre-condition to a merge to master.
-
developers are invited and very welcome to add to the unit tests and more configuration to the flow test to enhance coverage.
-
the analysts who deploy generally will not look at development snapshots.
-
there are px-dev machines, a cluster with some development configurations, that, I think Noureddine has set up to pull daily snapshots from master.
-
at some point, Peter decides a release is appropriate, and he goes through the procedure described in the Developers Guide to create it. This usually happens once or twice a month.
-
The following Wednesday after a new release is made, Noureddine has a procedure that updates hpfx machines (science and collab) to the latest release. Note: this is currently broken because we wanted an explicit manual install to cover the change amqplib--> amqp, so used the chance to make the package name more compliant. (python3-metpx-sarracenia -> metpx-sarracenia) So we have a one-time need for a manual install.
-
users on hpfx can try things out before the version goes further.
-
from that point analysts manually select the version to use on systems that are gradually more critical, more dev systems, more stage systems, and eventually operational systems.
-
It is typically months after a release is created that analysts doing operational deployments are comfortable. The operational deployments are typically meant for government-wide mission critical usage, and so caution is expected and prudent.
-
Once analysts are comfortable, they start recommending that version, and that is the answer sought to the which version is stable question.
There a major refactoring done at the end of 2017, basically completed by Jan. 2018, and releases after that point have essentially been bug-fixes*. Configurations for versions prior to that point have incompatibilities with >=2.18.01 (releases on or after January 2018.) so that the move to a current version needs care. Once at a recent version, all upgrades should be seamless. The only impact of upgrades should be getting bugfixes, and there is no reason for fear, but the analysts see the schism between before and after the re-factor, and it results in a great deal of caution.
In other words, the analysts don't necessarily believe the releases are just bugfixes, and the only thing that will convince is time, and making good releases. My guess is whatever investment we make in unit tests and the flow test will improve release quality and give analysts confidence in the released versions, but that takes time to prove.
ddsr*.cmc is the most critical cluster, and it is running a version from >= 2.18.10 (2018/October), having been upgraded at least once in 2018. So it is on the post 2018 bandwagon, and it since it has hundreds of configurations operating at very high volume, the version running there is likely the one we have the most confidence in.
For now, the only thing I can suggest is that we have a tag (that moves) and analysts vote on a stable version... we just move the pointer, as analysts opinions change. On the other hand, we have been in beta for a year or more, and at some point, we will probably just declare it stable and the releases should be rarer.
We would have to define what is considered stable for Sarracenia. Actually when the flow test is passing in one or many environment we consider (as an act of faith) that Sarracenia is stable.
But is it really the case ? Does it garantee that no user willl experience any major problem from their use case scenario ? What are those use case ? How many we support ? Is it posssible to list them all so we can control what is supported and what it is not ? If we can't, can we portrait our typical users from our major clients so we get such a list, then we may define what are the metrics and rules that define what would be a stable Sarracenia ?
Basically, the more we will have good and bad feedback from users that we analyse, consolidate into those functionnality, the more we will be able to measure which version is stable.
Actually, I'm sorry I only have unanswered question on that subject, and it will take for me more interactions with users to understand their real needs on that and to judge about what is stable.
Also what quality attribute Sarracenia is trying to fulfill in term of data exchange. Is it Reliability, Performance, Availability, Modularity, Testability... Also, how do we measure the goal we want to achieve with those and what are the priorities? After we will have a clear understanding on that, we will be set on whether our architecture is answering to all those needs and what needs to be fix at this higher level. Then we will know without a doubt that we have (or not) a stable version.
At least, this is how I learned to evaluate a project in software engineering, but this is not a one man job. Defining all this would involve the developper team, the management, the clients, the contributors, ... So, where do we start ?