Testing, verification and test automation

[monperrus]

Wikipedia references:

• https://en.wikipedia.org/wiki/Software_testing
• https://en.wikipedia.org/wiki/Unit_testing
• https://en.wikipedia.org/wiki/Regression_testing
• https://en.wikipedia.org/wiki/Code_coverage

"Automated testing is the act of conducting specific tests via automation (e.g. a set of regression tests) as opposed to conducting them manually, while test automation refers to automating the process of tracking and managing the different tests." source

[bbaudry]

Interesting tool. Detailed failure reports and hassle free assertions Scott test reporter https://github.com/dodie/scott

[monperrus]

From @AG00:

There is a relationship between desired properties (requirements), tests (verification) and test results. Those may be present in a hierarchy of components/subsystems/libraries. In addition, there is a relationship between planning activities on the system (product) being developed and its desired properties. Further, with time there will be more than one system/product configuration to consider and more than one version in use with different installations/deployments/customers. These relationships quickly becomes complex to understand and keep track of. As changes are being made (code or documents) and automatic tests are run, it would be good to have the information model changes generated rather than manually written. The information model can also be used to convince a customer (or the devops team itself) that a complex/high-level system property is achieved.

[bbaudry]

Testcontainers to test different environments on the developer's machine https://www.testcontainers.org/

[bbaudry]

A post about the challenges to get global metrics for code coverage https://massol.myxwiki.org/xwiki/bin/view/Blog/GlobalLocalCoverage

[bbaudry]

Security-based testing https://www.inspec.io/

[bbaudry]

DeFlaker: Automatically Detecting Flaky Tests https://www.jonbell.net/icse18-deflaker.pdf

[bbaudry]

Property-based testing in Java https://github.com/quicktheories/QuickTheories

[monperrus]

Cross Browser Testing, Selenium Testing, and Mobile Testing https://saucelabs.com/

[monperrus]

Perfecto - Cloud-based Devops Testing https://www.perfecto.io/

[monperrus]

Capturing test history https://stackoverflow.com/questions/37657130/how-to-capture-the-junit-test-result-of-project-in-database

[monperrus]

Automated Tool for Prioritizing JUnit Test Suites https://github.com/kanonizo/kanonizo

[bbaudry]

mutation testing https://github.com/hcoles/pitest

[bbaudry]

automatic unit test amplification https://github.com/STAMP-project/dspot https://arxiv.org/pdf/1811.08330.pdf

[bbaudry]

automatic detection of pseudo-tested methods https://github.com/STAMP-project/pitest-descartes https://arxiv.org/pdf/1807.05030.pdf

[bbaudry]

mutation testing in JS https://stryker-mutator.io/

[monperrus]

Property-based testing in OCaml https://github.com/c-cube/qcheck

@robertfeldt you may want to subscribe to this issue.

[monperrus]

Chris Parnin's Lecture on software bots https://docs.google.com/presentation/d/1JClXhqNX8ufzmxoDpVPCLRDlqu4tUkvbJgwYICw70kM/edit?usp=sharing

[bbaudry]

Testing of Microservices

https://labs.spotify.com/2018/01/11/testing-of-microservices/

[bbaudry]

Amazon codeguru It’s like having a distinguished engineer on call, 24x7 https://aws.amazon.com/codeguru/

[matsskoglunds]

Postman/Newman https://www.getpostman.com/

[bbaudry]

New testing course from TU Delft https://sttp.site/

[monperrus]

Rultor is a DevOps team assistant. It helps your programmers and release managers automate routine operations (merge, deploy and release), with an easy-to-use intuitive chatbot interface. https://github.com/yegor256/rultor

[bbaudry]

A series of recent posts about test and automation http://blog.testingcurator.com/2020/04/12/testing-bits-april-5th-april-11th-2020/

[monperrus]

ScalaTest: https://github.com/scalatest/scalatest

[monperrus]

Jasmine is a behavior-driven development framework for testing JavaScript code. https://jasmine.github.io/

[monperrus]

Catch2: modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++ https://github.com/catchorg/Catch2

[bbaudry]

Commonly used test data https://github.com/search?p=2&q=dQw4w9WgXcQ&type=Code

[monperrus]

Commercial product to accelerate tests with an automated dependency graph https://yourbase.io/

[bbaudry]

Automated Cucumber-JVM test pack for DECS built on the Serenity BDD framework https://github.com/UKHomeOffice/hocs-serenity-automation

[bbaudry]

Testing at Electronic Arts SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning https://arxiv.org/pdf/2203.05566.pdf

[bbaudry]

Property-based testing in language X For example, in javascript https://marmelab.com/blog/2019/04/18/property-based-testing-js.html

[monperrus]

An interesting essay / demo / tutorial topic: Automated migration of Junit4 to Junit5

[monperrus]

Suggestion: Testing for GraphQL APIs and Engines.

[bbaudry]

An interesting and very original topic for a presentation / demo in software testing: code coverage in production

[monperrus]

Topic suggestion: in depth analysis of a mocking library in language X (eg Mockito)

[bbaudry]

fuzz testing is an essential topic for testing and security

[monperrus]

Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation (RPA). https://robotframework.org/

[bbaudry]

Predictive Test Selection cuts testing time up to 70% by using machine learning to select and run only tests that are likely to provide useful feedback during test runs https://gradle.com/gradle-enterprise-solutions/predictive-test-selection/

[bbaudry]

Load-Test-As-Code. https://gatling.io/

[bbaudry]

Instancio is a Java library that automatically creates and populates objects for your unit tests. https://github.com/instancio/instancio

[monperrus]

Mimicking Production Behavior with Generated Mocks 2022

[monperrus]

Mutagen and cargo-mutants are two relevant mutation testing tools for Rust.

Credits @VIGGEEN cc/ @ErikNatanael

[monperrus]

following up on Jepsen JepREST: Functional tests for distributed REST applications.

[bbaudry]

A command-line benchmarking tool for rust https://github.com/sharkdp/hyperfine

[bbaudry]

BDD with Rust https://github.com/cucumber-rs/cucumber

[bbaudry]

A library for generating fake data in Rust. https://github.com/cksac/fake-rs

[bbaudry]

The art and craft of test-driven development (Increment, 2019)

[monperrus]

Integrity Test Framework - a sophisticated integration test automation framework for Java https://github.com/integrity-tf/integrity

[monperrus]

Newman is a command-line collection runner for Postman https://github.com/postmanlabs/newman

[bbaudry]

Framework for web testing, and capture&replay https://github.com/microsoft/playwright