Definition of "scalability" vs. "elasticity testing"

[samm82]

Scalability Testing

SWEBOK V4 says "scalability testing evaluates the capability to use and learn the system and the user documentation. It also focuses on the system’s effectiveness in supporting user tasks and the ability to recover from user errors" (p. 5-9). This seems to be an amalgamation of usability, functional?, and recovery testing. (Hopefully the formatting helps and doesn't make things more confusing!) This seems particularly odd when an earlier subsection of Software Testing gives two definitions of scalability:

1. "The software’s ability to increase and scale up on its nonfunctional requirements, such as load, number of transactions, and volume of data"
2. "Connected to the complexity of the platform and environment in which the program runs, such as distributed, wireless networks and virtualized environments, large-scale clusters, and mobile clouds" (p. 5-5); I'd paraphrase this as "the ability of the system to be built upon"

Cross-checking this definition in other sources:

• IEEE2022 mentions scalability testing (as a specialized form of testing of Internet of things systems), but doesn't define it
• ISO/IEC 2023 defines "scalability" as the "capability of a product to handle growing or shrinking workloads or to adapt its capacity to handle variability" - primarily the first definition above
• ISTQB defines "scalability testing" in terms of "scalability", which it defines as "the degree to which a component or system can be adjusted for changing" - a combination of the above two definitions

Elasticity Testing

However, SWEBOK V4 also says "elasticity testing assesses the ability of the SUT … to rapidly expand or shrink compute, memory, and storage resources without compromising the capacity to meet peak utilization" (p. 5-9). From a cursory Google, this seems to track, and I also remember there was a distinction between the two from my Microservices courses (but not what it is 😬). This definition goes on to say that an "elasticity testing objective [is] … to evaluate scalability" (p. 5-9). This implies that elasticity testing is a parent of scalability testing, but the given definition of "scalability testing" doesn't seem to suggest this.

Wanted to make sure that there isn't some nuance I'm missing before I accuse SWEBOK of being wrong! 😅

Spike Testing

This is "testing to determine the ability of a system to recover from sudden bursts of peak loads and return to a steady state" (Hamburg and Mogyorodi, 2024), which may be relevant to this discussion.

[JacquesCarette]

This feels like a typo in SWEBOK V4. They give the definition usability (or learnability) but tag it scalability. The Software Testing ones track.

Most scalability testing is about "scaling up"; it makes sense to me to talk about "elasticity" as well - i.e. is your memory (etc) footprint back down to something small after you've done something large, but now no longer are?

[smiths]

I agree with @JacquesCarette. @samm82 you've found another problem with the existing publications on testing-related definitions. Your list is getting longer and longer! :smile:

[samm82]

This comment really contains two separate points about scalability, specifically its definition compared to other properties.

1. vs. Elasticity

ISO/IEC TS 25052-1:2022 defines "scalability" as the "degree to which physical and virtual resources are available automatically and immediately, when they are needed, subject to constraints of service agreements" and "elasticity" as the "degree to which a cloud service adjusts rapidly and elastically the amount of resources that are allocated to an instance of the service". Likewise, a textbook cited by SWEBOK V4 for its "definition" defines "scalability" as "the ability of the system to cope with increasing numbers of users without reducing the overall quality of service that is delivered to any user" (p. 515), i.e., "scaling out" (p. 516), but doesn't define "elasticity" (although I guess it would be implied to then be "scaling up" (p. 516) but also "scaling down"), despite SWEBOK also citing it for its definition of "elasticity testing".

Although the definition of "elasticity" is a little narrow, I think these two sources give a clearer distinction between the two terms and seem familiar to me from my microservices course (I guess I could always double check with Dr. Mosser! 😅). This definition is also consistent with SWEBOK V4's definition of "elasticity testing".

2. vs. Expandability

I also think "scalability testing" should focus more on SWEBOK V4's first definition of "scalability", since the second definition seems more like "expandability" (although ISO 20887:2020 also describes it as "a form of scalability") - this means that scalability is a parent of elasticity, as opposed to the other way around, suggested by SWEBOK V4's definitions.

[smiths]

Good job @samm82 with summarizing the different definitions from different sources. I don't love that the definition of elasticity uses elasticity. I can see the difference between the definitions of scalability and elasticity, but I wonder if the distinction is practically relevant. If we have the term scalability, do we really need a similar concept of elasticity?

[samm82]

I'll dip back into my Microservices notes and see if there's a difference between "scaling out" and "scaling up".

Based on our discussion, scalability is about scaling up to adapt to increased demand, where elasticity is about scaling down (after having scaled up) to adapt to decreased demand. Would this mean that elasticity testing is a parent of scalability testing? If elasticity testing is performed (correctly), would scalability testing be worthwhile?

[smiths]

I found another source of information on scalability versus elasticity. This source isn't academic, but the definitions make sense based on our earlier conversation. "Scalability addresses the challenge of maintaining consistent growth of your system over time, while elasticity addresses the problem of unexpected spikes that can occur in your system during regular operations." With the "time" component, I think the terms aren't just about growing and shrinking. If an algorithm is O(n!) it isn't going to scale. I'm not sure either scalability or elasticity is a parent of the other. Scalability is about growth over time. Elasticity is about dynamically growing and shrinking. It feels more like scalability and elasticity are children of nonfunctional testing. There might be another category that is more relevant. If we are going to have a hierarchy of testing techniques, we should make sure that it is a useful hierarchy. We probably don't want to get into premature classification of the techniques.

[samm82]

Agreed. This also distinguishes "scaling out" and "scaling up" to me: "scaling out" being an increase of the max capacity, where "scaling up" is an increase of the current capacity (obviously accompanied by "scaling down"). Since they are now "separate", they wouldn't have a child-parent relation to each other. Thanks!