Vinai
commented
7 years ago They are fundamentally different.
MTF is a functional testing framework, that is, the browser is remote controlled via Selenium. Functional tests are black box tests, meaning they assume no knowledge about the implementation.
This repository contains an integration testing framework.
Integration tests are white box tests, meaning the tests have knowledge about the implementations, that is, the PHP classes and methods.
Integration tests confirm the PHP classes work together as expected.
There also are unit tests, which usually do not require any testing framework, since they test a single class in isolation.
There also are many other types of tests, but to answer your question these three should suffice.
Functional tests, integration tests and unit tests all test the application on a different level. They are not mutually exclusive, but rather build on top each other.
The classical testing pyramid for a project contains a large number of unit tests (maybe a couple of thousands), a smaller number of integration tests (a few dozen to a few hundred maybe), and a handful of functional tests (maybe up to 10).
That said, functional tests are problematic. They take the most effort to write and maintain, and take the longest time to run. Because of the complex stack they tend to be flaky, meaning tests sometime fail for no apparent reason.
Many organizations have abandoned functional testing because of that, since a good unit and integration test coverage can give enough confidence to implement continuous delivery.
If you are just starting out with testing, I would recommend focussing on integration testing. Then the MTF can be used to add a few acceptance tests that are important to the business, but use integration and unit tests during development.
If you want to read further, here is an interesting blog post by google giving more details why they stopped using functional tests: https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html
If you still want to continue down the functional testing path, I would recommend against using the MTF and instead use plain PHPUnit and a tool like phpunit-mink, or codeception instead of phpunit, because the MTF is dreadfully complex and tests are very abstract and hard to read.
I never used PhpUnit but I know Magento 1.9.x integrate MTF that is based on that.
Why should I prefer this framework to MTF ? Do they have different aim/scope or this it is just a different approach to integrate PHPUnit in Magento ?