2010/04/13/automated-testing-is-more-than-automated-checking

[utterances-bot]

Automated testing is more than automated checking | Arjan’s take on Testing

I was involved in an interesting discussion on Twitter, the other day. It started with a tweet by @testingqa (Guy Mason): “Still of the opinion that ‘Automated Testing’ is a deceptive term,no testing is being performed,it should be called ‘Automated Checking’ #qa“. With that he probably refered to Michael Bolton’s blog that there is a difference between testing and checking.

https://arjankranenburg.github.io/2010/04/13/automated-testing-is-more-than-automated-checking.html

[blog-visitor]

Michael Bolton (2010-04-14 AT 1:32 PM) Hi, Arjan…

Thank you for pointing people to the blog series.

I agree that automated checking doesn’t cover the activities done in automat[ion assis]ted testing, and I agree with your admonition not to call the latter by the name of the former. At the same time, I also counsel the opposite too: I observe that some individuals and groups become fascinated with or dazzled by the checking, and forget the important of the testing activities that surround it. I try to emphasize this in http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/.

Again, thank you for the link.

—Michael B.

[blog-visitor]

Albert Gareev (2010-04-15 AT 1:33 PM) Hi there!

I’m not an “automate everything” enthusiast neither am I an exploratory testing crusader. I stand for a systems thinking approach.

The first thing to keep in mind while reading “Testing vs. Checking” series is a kind of automation talked about. It is unit testing, mainly with regards to TDD approach. Sure, an executable process which is essentially a calling of a function with a bunch of inputs and checking results it returns perfectly fits definition of automated checking. However, it’s a shame to downgrade unit testing activities to unit checking only. Code review is as much sapient activity as exploratory testing.

I tried to emphasize that aspect in one of the questions I asked.

The main question, that wasn’t answered, is also – why compare code testing with product testing? While sapient testing skills can be (should be) applied at any level, call-checking of code components pursues a different goal than testing of application’s functionalities. One does not substitute another.

Opposition towards using automated tools is called Ludditism. Yes, machinery threatens to scripted types of jobs. For others, it helps increasing productivity or reaching new levels of exploration.

Tools help in different areas. Sometimes they do not substitute manual activity: you can cut one’s body to see what’s inside or you can X-Ray it. Just keep in mind that a tool doesn’t actually see (i.e. perceive) – it brings a picture for a human being to perceive.

You can manually dig an archeological site or you can use electromagnetic scanning and build a 3D picture with a computer. Once again, keep in mind, a program won’t care what is on display.

In software testing, activities where engagement of automation tools helps reaching new levels would be simulated load testing or certain kinds of security testing.

In automation of GUI activities, recording/checking tools occupy a thin niche of regression testing approach, based on the heuristic “if a functionality is not broken then the test passed last time will pass again”. But this heuristic is not really reliable, it doesn’t work too often.

Thank you, Albert Gareev

http://automation-beyond.com/

[blog-visitor]

John Stevenson (2010-04-16 AT 1:34 PM) Hi Arjan

Do not know what happened to my last comment but I will try again

After reading through your article again I can see the point you are making.

I do have a problem where automation is used to ‘test’ a product and a large amount of time and effort is spent putting together a framework for the automation, writing the automated code and then running the automated code.

The actual creation of the automated ‘tests’ requires interaction with a human the running of these ‘checks’ does not. They become static checks not dynamic tests. What then normally happens is the software is changed, the tester then needs to change the automated suite to match the changes in the software instead of looking at the software and exploring what is happening.

IMO automation is valuable as a tool to be used when carrying out testing (exploratory). Automation can be used to:

Create data sets

Set up a test environment

Carry out repetitive steps than must be done so testing can b carried out.

I do love your term ‘Automation Assisted Testing’ and I might have to borrow that (acknowledging source of course) I see automation as a useful tool to aid testing I do not see it as a replacement for testing.

I hope this makes sense since I have had to re-write it.

regards

John

[ArjanKranenburg]

(2010-04-17 AT 1:37 PM) Thanks for your comments.

Just to make this clear, the term Automation Assisted Testing is not my ‘invention’. Credit where credit’s due, I believe it was either Michael Bolton or Adam Knight who came up with the term.

[blog-visitor]

Michael Bolton (2010-04-18 AT 1:46 PM) A couple of points:

1) There is such a thing as automated checking; it is the checking that is done by automation (surprised by that?). That is: an observation and a decision rule linked to that observation, such that both can be performed non-sapiently, without human observation, judgement, wisdom, or engagement. Now: that can’t happen without (human) testing activity preceding it, and it is of very questionable value without (human) testing activity following it, but the automated checking part does happen. The mistake that many people make is to make a big deal about the checking part (10,000 automated tests!), and not paying attention to the testing stuff. What quality criteria do the checks cover? What oracles (principles or mechanisms by which the checks help to identify the problem) are being used? What are the context drivers for choosing when and how to implement a particular set of checks? How will the information be reported? Is the testing approach sufficiently diversified? and so forth. Those questions can neither be asked nor answered by automation. Testing rules!

2) I appreciate Arjan giving me credit for the notion of automation-assisted testing, but I’m skeptical that it’s attributable to me. It sounds more like Cem Kaner, James Bach, or Bret Pettichord. In the unlikely event that I was actually the first to utter the words in that order, it’s because they inspired me to do it.

—Michael B.

[ArjanKranenburg]

(2010-04-19 AT 1:48 PM) Hi Michael,

Thanks for your comment.

The number of automated checks, scripts, or cases is not a big deal, but having test automation can be of big value. Like every automation, it should support you in doing your tasks more effectively and efficiently. Like every automation it can be too much in the sense that the costs do not outweigh the benefits and like every automation it remains to be seen if the investment will pay off.

When projects consist of short iterations and increments, more and more these days, the tedious repetitiveness of actions and checks is a sure candidate for automation. Especially tests on method level and components can be useful to automate. As a hobby-programmer, I know it is easy to break something. Having a compiler or test-suite telling you within seconds that something is wrong, or at least different, is of great value in developing software.

The tester can finalize those tests by going quickly through the results and then put all his/her energy in the interesting parts, e.g. the problem areas, the new functionalities, and/or tests on a more broader scope.

But indeed: Testing Rules! Test Automation only serves it!

[blog-visitor]

Ray Oei (2010-04-20 AT 1:50 PM) Hi Arjen (and Michael),

The first time I saw the term ‘Computer Assisted Testing’, which resembles your ‘Computation Assisted Testing’ was when I read “Architectures of Test Automation” from Kem Kaner (http://www.kaner.com/pdfs/testarch.pdf). I really liked the ‘done by humans’ part in it 😉

He some other interesting thought regarding Computer Assisted Testing.

Regards,

Ray

[blog-visitor]

Michael Bolton (2010-07-21 AT 1:49 PM) After all this time, there’s something else worth pointing out: checking is often automated, but there is non-automated checking as well. Some people ask humans to check, that is, to follow a specific set of instructions and, basically, act like machines. That’s manual checking. Automated checking therefore

—Michael B.