Distinction between a "technique" and a "class/family of techniques"

[samm82]

In looking through IEEE2021, I noticed that it seemed to classify groups of techniques (or even types) as practices: “experience-based testing practices like exploratory testing … are not … techniques for designing test cases”, although they “can use … test techniques” (IEEE, 2021, p. viii). This is a consequence of #26, and might not even be worth investigating in-depth at this point, since we might land on another classification scheme, but I thought I would note it here. Any thoughts?

Full explanation from my notes (footnote is part of it)

> …It originally seemed like experience-based testing was categorized as both a test design technique and a test practice on the same page (ISO/IEC and IEEE, 2022, p. 22)! This also caused confusion about its children[^1], such as error guessing and exploratory testing; error guessing is an “experience-based test design technique” and “experience-based test practices include … exploratory testing, tours, attacks, and checklist-based testing” (p. 34). However, these authors previously say “experience-based testing practices like exploratory testing … are not … techniques for designing test cases”, although they “can use … test techniques” (2021, p. viii). This implies that “experience-based test design techniques” are techniques used by the practice of experience-based testing, not that experience-based testing is itself a test technique. If this is the case, it is not always clearly articulated (ISO/IEC and IEEE, 2022, pp. 4, 22; 2021, p. 4; Washizaki, 2024, p. 5-13; Hamburg and Mogyorodi, 2024) and is sometimes contradicted (Firesmith, 2015, p. 46). A similar issue exists with “model-based testing”, which is sometimes called a practice (ISO/IEC and IEEE, 2022, p. 22; 2021, p. viii) and sometimes a technique (Kam, 2008, p. 4; implied by ISO/IEC and IEEE, 2021, p. 7 and 2017, p. 469). > Furthermore, if a “class of … techniques” is a practice, then other “techniques”, such as combinatorial testing (ISO/IEC and IEEE, 2022, pp. 3, 22; 2021, p. 2; Washizaki, 2024, p. 5-11; Hamburg and Mogyorodi, 2024), data flow testing (ISO/IEC and IEEE, 2022, p. 22; 2021, p. 3; Washizaki, 2024, p. 5-13; Kam, 2008, p. 43), performance(-related) testing (ISO/IEC and IEEE, 2021, p. 38; Moghadam, 2019, p. 1187), and security testing (ISO/IEC and IEEE, 2021, p. 40) may also actually be practices, since they are also described as classes or families of techniques. The same could be said of the more general specification- and structure-based testing, especially since these, plus experience-based testing, are described as “complementary” (p. 8, Fig. 2). [^1]: I had originally noted that this may suggest that parent and child test approach categorizations aren’t necessarily the same, or that more thought must be given to classification/organization; this may still be true, since there are several other instances of this inconsistency.

[JacquesCarette]

You should come up with a definition (for technique and family of techniques and practices) that we (@smiths and I) agree with, and then see where this falls. These words are, informally, so fuzzy, that it's impossible to really tell!

[samm82]

These are the definitions I have so far, from Table 2.1 in my notes:

• Technique: A defined and systematic procedure used to create or select a test model, identify test coverage items, and derive corresponding test cases that generate evidence that test item requirements have been met or that defects are present in a test item; a variety is typically required to suitably cover any system and is often selected based on team skills and familiarity, on the format of the test basis, and on expectations.
• Practice: A conceptual framework that can be applied to a test process to facilitate testing; more generally, a specific type of activity that contributes to the execution of a process.

Paraphrasing, a "technique" is a method for deriving a given test case and a "practice" is a method for actually performing it (e.g., branch testing vs. scripted testing).

So far, it doesn't seem like there is a "definition" for "family of techniques"; it just seems to be used as a modifier as opposed to a formal categorization.

[JacquesCarette]

This makes sense to me.

I'd say a "family" occurs when there are commonalities and well-identified variabilities that can be instantiated. As long as the commonalities are large and the variabilities smaller, it makes sense to group things together into a 'family'.

[smiths]

Nice definitions @samm82. I really like the traceability to the source of the definitions and your mention of synonyms and examples. Test is a very overloaded word!

I agree with the definition of families suggested by @JacquesCarette. If we put things together in a family, we should at least have a rough idea of the commonalities and variabilities (and the parameters of variation for the variabilities).

@samm82 is your question answered? Can we close this issue?

[samm82]

I agree that a "family" or "class" of testing approaches, whatever their categorization is, is meaningful. However, my main discussion point is what to call these.

I think I failed to include a crucial bit of information: despite the claim that "experience-based testing practices like exploratory testing … are not … techniques for designing test cases" (IEEE, 2021, p. viii), "experience-based testing" is later defined as a "class of test case design techniques" that includes error guessing (p. 4). From this, I wondered if a class/family of techniques could be considered a practice.

The terms "practice" and "technique" are pretty well-defined (at least in my mind), as shown above. The ambiguity comes from this new idea of a "class of techniques". If a practice is "a conceptual framework that can be applied to a test process to facilitate testing", could this include the use of a family of techniques? This seems to be what the literature implies (see the above quote), but doesn't seem meaningful as it reduces the definition of "practice" to "something done to test", which doesn't really distinguish it from "approach" (i.e., almost everything in my table could then be called a "practice").

Even just writing this out seems to indicate to me that the ideas of "practice" and "technique" should be kept separate. Does this make sense?

[smiths]

@samm82 rather than explain the apparent inconsistency in the IEEE definitions by interpreting a "class of techniques" as a practice, I think we should instead say it is an inconsistency in the IEEE definitions. (Another example for you to write about!) They first say that exploratory testing is not a technique, but later say it is a technique. This seems like a mistake to me.

I agree that technique and practice should be kept as separate. The distinctions seem useful.

The term family should also be kept separate. I think we can have a family of testing techniques and a family of practices. The definition of a family is the one that @JacquesCarette gave above.

I believe you are using class as a synonym for family. Is this intentional?

[samm82]

@smiths I made this change in fc51c61 (and will add more categorization discrepancies later to analyze); if you (and/or @JacquesCarette) like those changes, then I think this issue can be closed! 😁

[smiths]

It is too hard to review the diff from the commit. I'm not sure what I'm supposed to be looking for. :smile: Can you explain the changes that you want us to review?

[samm82]

My bad! 😅 The difference is essentially just Section 2.3.1 Categories of Testing Approaches in my notes.pdf

[smiths]

The notes section looks good to me @samm82. I may have different feedback when I read it in context, but going through it now I really like the traceability and connections to the existing documents/standards. Nice!