Testing support

[qiangxue]

Better support for testing models and controllers.

[andersonamuller]

What about using this: http://codeception.com/

It supports the current version of Yii

[samdark]

Can be a good idea.

[yiqing-95]

codeception is cool :+1:

[zeeke]

It could be very useful to have codeception working with yii just out of the box.

2013/4/2 yiqing-95 notifications@github.com

codeception is cool [image: :+1:]

— Reply to this email directly or view it on GitHubhttps://github.com/yiisoft/yii2/issues/45#issuecomment-15764114 .

[DavertMik]

+1 for Yii2 in Codeception =)

What should be done for smooth functional testing:

• configuration environment for testing (preinstalled, isolated)
• wrappers over session/headers/cookies functions, so we could mock them
• transactional fixtures in ORM (i.e. to clean them on rollback)
• various test helpers (to login/logout user)
• memory control. Simple cleanup of all globally stored variables.

Author of Yii1 module for Codeception, Ragazzo may share his experience on integration. I hope Yii2 will get a nice testing design :+1:

[qiangxue]

@DavertMik very nice summary. Exactly what I'm looking for. If you have more requirements, please feel free to add them.

Can anybody explain why we should use Codeception over PHPUnit? Note that we also need to consider the user base when choosing a technology.

[ghost]

Talking about the user base: If Yii wants to focus on BDD then you might also want to consider using Behat. Both "frameworks" seem to be conceptually similar (I am not familiar with Codeception though) but Behat seems to have the bigger user base right now.

[andersonamuller]

Taken out of codeception site:

Question: Why should I use this instead of PHPUnit?

Answer: Being the most popular unit testing framework PHPUnit has very limited features for functional testing with Selenium or other backends. Codeception is PHPUnit on steroids. Everything you need for testing is built-in and works just out of the box. No more pain in configuring Selenium, data cleanup, writing XPaths, and fixtures.

Question: No need for this. It looks just like Behat

Answer: Hold on! Even though it uses the same style in test descriptions and Mink as a driver, Codeception is completely different. As you know, Behat is not a testing framework, but Codeception is. Codeception tests are written in PHP they are written much faster, especially if you use an IDE. You can use variables and operators in your tests. You can use CSS and XPathselectors in your tests. These features allow you to build a solid test automation platform for testing your web application. Codeception tests are very readable for your developers, managers, and QA team.

[Ragazzo]

@qiangxue codeception can run phpunit tests too. i think supported tests base must be as this: phpunit (for unit-tests), codeception as a functional and acceptance tool out-of-the-box(there are some problems with current 1.1 yii integration but anyway), and maybe behat for those users who are in-love with BDD (i mean support behat by itself, because yes it is like "true" BDD), so do u think is it possible? If not and need to chose only one thing, i think it must be a codeception (u can look inside current Yii1.1 implementation in the codeception repo).

[Ragazzo]

@Haensel i think codeception user base is growing too, maybe not too fast because not so many people know about codeception, but is growing anyway ))

[DavertMik]

@qiangxue Actually Codeception is oriented towards functional/acceptance testing. But it uses PHPUnit engine for test execution. I.e. all PHPUnit tests can be executed in Codeception natively.

Cons: Not all PHPUnit infrastructure is ported to Codeception (groups, dependent tests, work with data, data providers, paralel test execution). Pros: there are basic yet useful helpers over PHPUnit. Check data in database, for example. Pros: functional tests and unit tests are executed in one engine with overall codecoverage report.

[tonydspaniard]

@qiangxue Would this testing framework implement frontend testing platforms too (ie. jasmine, testacular, qunit, etc?) I know that Yii Framework is a PHP oriented library, but I believe there is a missing link at the frontend level.

In my humble experience with Yii, I found my self on the need to implement spectacular js libraries for tasks automation (GruntJS), MVV (BackboneJS), and so on... Javascript is here to stay and would be great if we could find the best way to integrate Yii 2 to easily work in combination with these libraries.

[tonydspaniard]

By the way @Ragazzo, thanks for the hint with Codeception. We are using it on a very big project and works nicely -there were some bugs though, but the team submitted the patches.

[Ragazzo]

@tonydspaniard all thanks to @DavertMik ) btw. your team submitted patches to the codeception repo or u mean you patched sum bugs in your own build or smth. like that (anyway thanks for feedback =) )? I also think that Antonio is right, some testing support on js would be very great.

[nurcahyo]

"better support in testing controller" it is mean we can use code-coverage testing for controller in Yii2? :smile:

Using codeception is good Idea for functional testing, but i think it would be very usefull if we can use standard PHPUnit testing too so user can make own choice to use PHPUnit or codeception.

[ghost]

I've migrated from "vanilla PhpUnit" to Codeception and my tests are still the same (at least the assertions). You can use all the standard assertions of PhpUnit and don't have to use any of Codeception's features.

[DavertMik]

@nurcahyo When you use

XyZTest extends PHPUnit_Framework_TestCase

this test can be executed in the same manner in Codeception and PHPUnit. Maybe some running options are different.

[Ragazzo]

any updates on this one? as i remember there were a lot of "+1 for Codeception" here, but now they are gone, maybe they were deleted by someone or one of the github nodes has gone, dont know)) but will this one be adopted before "alpha"?

[qiangxue]

I deleted them as we are aware of the codeception option. We'd like to see more constructive advices regarding using codeception if you have experience with it. Thanks.

[Ragazzo]

ok, what reasons i need to write here for adoption (some codeception vs phpunit+selenium/phpunit or what?)? also other developers that use codeception vote for that two, i dont think that it is good to just throw their opinions away :S

[samdark]

These were not argumented opinions. Just +1s. We know that people are using Codeception but we're interested in how and why it's a good option when compared to other solutions: behat, selenium etc.

[Ragazzo]

Ok, but do i need to compare codeception vs behat+mink + phpunit+selenium ? will this be ok?

[DavertMik]

@samdark Let's continue this discussion in reality on HotCode conerence, ok? We may postpone this question by that time. I hope I can convince you :)

@qiangxue Just for you to notice, we did not encourage people to post +1, they did it on their own will.

[Ragazzo]

Well, not so wonderful as it can be)) but i will also provide some article (writing it now) on how i use codeception with Yii, and i will add some compare between codeception/behat+mink/phpunit+selenium, will this be good @samdark ?

[samdark]

@DavertMik discussing it at HotCode is a very good idea.

I've worked with these two @Ragazzo mentioned but have almost no experience with codeception. It looks very similar to behat.

If compared with selenium codeception pro is that it provides more stable solution for simple functional testing like behat + mink does. Syntactic sugar is nice as well but not as important, I think.

[Ragazzo]

Well the main difference between codeception for acceptance and functional tests and behat+mink is that behat is more like feature-driven-development, i mean that all features must be executed in one context (u can attach subcontexts also but it is not a full solution) this leads to some errors when you have some same strings in your scenario and this will throw exception because of the same php-docs that are parsed by behat, also it is difficult as i pointed above to split features in separated classes (php class features), in codeception it all is done in easy way, you can write similar flat-scenarios with php, or u can use classes and make some PageObject pattern with them, and codeception also provides high-level-api if i may say so for mink, so user do not need to be afraid of that if he has 20-30 tests that they will have same methods names as i would be in behat. Also need to mention that codeception is a fullstack framework, it has about 20 modules for REST, XML-RPC, Db and other, and behat has no. If compare codeception with phpunit+selenium the main advantage for codeception is that it can run different type of tests (functional and acceptance), but phpunit+selenium are only acceptance tests, and also with mink codeception can use different drivers, but phpunit only use selenium :S well this is a short list of differences between this three. Also i need to notice that behat is very good for end-to-end tests and for developing in that way: BDD->TDD(unit-tests).

[DavertMik]

But regarding the @samdark comment I need to explain a bit more. Why not Behat+Mink or Phpunit-selenium

Let's distinguish functional and acceptance testing (we did that in Codeception).

Acceptance testing has absolutely nothing to do with framework. When you execute PHPUnit + Selenium your test would work through phpunit>selenium>browser>webserver>application. When you test with PhpUnit + Mink, you may have this option: phpunit>curl >webserver>application. In both cases we hit web server. Thus, all internals of application are hidden from testers.

Cons:

• Tests run slower, as they run through web server and browser.
• Tests does not show us the actual errors and stack traces.

In functional testsing you emulate browser in the framework environment. In this case you have only: codeception > application. I.e. you don't hit a web server, browser, curl, or anything else. If you take a look on other frameworks: Laravel4, Symfony2, Zend2, they already have functional testing support.

So you can test user can log in/log out/create articles/delete articles, without launching a web server. Just by emulating GET/POST and matching the response. In my comment above I described what actions should be taken to make this functionality to work smoothly.

If Yii2 will have support for functional testing there would be no matter of testing framework it will use. It may be Codeception, PhpUnit, Behat, etc.

[jacmoe]

One thing that continually trips up a lot of people using testing with Yii is the configuration of the PHPUnit Selenium 'module'.

If Codeception does that for us, that is going to be a winner.

Personally, I am not a bit fan of too much syntactic sugar, but I guess I can live with that.. Codeception seems to take syntactic sugar to a whole new level. If Codeception can bring Selenium powered functional testing to Yii in an easy way, then I'd say: go for it. It seems to be powered by both PHPUnit and Selenium under the hood, right?

[samdark]

@jacmoe you can use regular phpunit syntax with codeception, as I know.

[Ragazzo]

@jacmoe you need about 2 minutes to make your functional/acceptance tests work with codeception, really. also you can run your phpunit tests there as well. codeception provide different drivers (this is advantage of mink of course) not only selenium2.

[jacmoe]

@Ragazzo Really cool! I think I'm sold on Codeception already - it means that we can use PHP and what we know from PHPUnit and leave the grunt work to Codeception. I think that it can actually compel me to actually start using proper testing methodology. :) Since Codeception seems to be extremely flexible, I think it would be very Yii-ish to make use of it.

[Ragazzo]

@jacmoe i think this is my last post on this topic (really looks like bumping my messages here))) ) so i also need to notice that main-core-developer is @DavertMik and not me) and that it is better to use codeception for functional/acceptances and other tests maybe api tests (REST/XMLRPC) and other, so as for me i stay on unit-tests with phpunit (not functional with selenium) also you can run them with codeception, but i use codeception for functional/acceptance tests.

[jacmoe]

What really prevents me from doing full out testing with Yii is that it is difficult (and cumbersome) to move beyond unit testing, and I think Codeception really can help there. Thank you for bumping the issue @Ragazzo - if anyone should be accused of undue posting, it would be me. ;p I just think that if Yii goes the Codeception route, it would mean that testing with Yii could be a unified and wholesome experience. And it would definitely cut down on the support requests in the Yii forum!

And, Behat? It feels strange and magical, and not in a good way. I very much prefer to express my intentions in PHP.

[edit] If we should not have discussions in the comment section, what is the point of having it? I think it's important to discuss.

[Ragazzo]

Well, behat by itself (without mink) is just a "true" story BDD, when codeception is bdd-story-syntax-written fullstack framework) my opinion is just to use right tools for right things, thats all) aaand i am out))

[qiangxue]

Thank you all for these informative discussions!

@Ragazzo would it be possible for you to show me some use cases with codeception? In particular, I'd like to see what you want to test in a typical project, what test code you write, and what test support you would like to have from Yii. If you could help identify what kind of classes are needed, it would be even better. Thanks!

[Ragazzo]

ok, i am writing article now about how i use codeception (with new cest-classes format for functional and acceptance tests, and integration with Jenkins), i will finish it in a few days, i will add there some compare with other tools and how to use codeception. Also all that need to be done is not so big, you also can contact @DavertMik, some common pitfails like dumping headers/sessions_start calls and other i will notice in this issue after i will publish article. Also maybe it would be a good way to create new topic on yii-en forum where i can write all this pitfails and classes as you asked?

p.s. also some good article for those who already read some of the guides of codeception, http://phpmaster.com/ruling-the-swarm-of-tests-with-codeception/. Article is written by the main core-developer.

[qiangxue]

Thank you, and take your time with the article. I will certainly bug @DavertMik and you more. :+1:

[klimov-paul]

For me "Behat / Jenkins" is more then just a testing tool. It is an entire "Continious Integration" solution, which allows code delivery, branches/tag managament and so on. Unfortunally I am not familiar with codeception, so I can not tell anything about it and thus can not compare it with "Behat / Jenkins". "Behat / Jenkins" has a plugins and extensions, whihc allows to run tests for SOAP and RESFULL API and other complicated things.

[klimov-paul]

"Behat / Jenkins" syntax has been created to allow unexpirienced users to write a tests. Once my company worked on the project when the customer specified this requirements via tests on "Jenkins" and then we wrote code to satisfy it.

[klimov-paul]

My opinion: the program should be designed in the way in could be tested via unit test only. At least this is valid for the PHP part of the application. I suppose we should concentrate on the matter to create enough “test gaps” in the Yii core, so it could allow writing unit tests for the most possible scenarios and features. For example: I should be able to easily check if the correct controller action was invoked for particular request data, I should be able to mock database operations (sometimes it is enough to check the data insertion has been invoked and not necessary to actually insert row into database), I should be able to mock Active Record classes (which is impossible in Yii 1.1.x) and so on.

For me Yii core should support the PHPUnit and its official extensions, because this is definitely a standard in the PHP code testing. Anything beyond this is an additional feature.

While setup “Jenkins” in my company we have no need to modify or extend Yii core. I actually do not really understand why we should bind the Yii 2 to any particular BDD or DDD technology? What modification actually should be performed to enable Yii 2 compatible with codeception?

[Ragazzo]

@klimov-paul what do u mean by "Behat / Jenkins", here we talking about not integrations and just testing? Any integration with CI can be made by simple junit and some Jenkins plugins. There will be small modifications, why do you voting against if you are not maybe familiar with codeception or behat by themselves, no offense really of course ?

[klimov-paul]

what do u mean by "Behat / Jenkins"

Sorry my bad. I have worked so long with the Jenkins + Behat integration, so merely forgot where is division line. Hudson / Jenkins is an continious integration tool (very powerfull actualy), which commonly uses behat as testing support.

why do you voting against

I am voting for the unit test suport most of all, assuming that functional testing could be performed by particular developer using technology he think appropriate.

[Ragazzo]

well, looks like some misunderstanding from my side, the current situations is to :

1. for classical unit-tests use phpunit (i also prefer to use it) . This is not under discussion.
2. using codception for acceptance/functional tests (this is what we are discussing). So if vs this two, then not codeception vs phpunit, just codeception(with different drivers Goutte, Selenium1,2, ZombieJs, Sahi, ... almost all mink drivers) vs phpunit+selenium1,2.

[klimov-paul]

I still remember the time (short period actually), when I worked on the desktop application development. I remember writing unit test for the application GUI on Java without(!) any additional test frameworks. The GUI classes were designed in the way, they can be tested with the simple asserts (even without a unit test framework). Buttons were able to emulate clicking, edit fields allowed to be filled with test data and so on. For me such ability is an ideal, which is worth to be achieved.

Controllers and models (see the issue title) should be testable with unit tests!

[klimov-paul]

I have nothing against functional testing tools: Selenium, Behat, Codeception and so on. I think we should not bind to the specific technology, because it could not cover everyone needs. Selenium support is an official PHPUnit extension, so I can accept its integration, besides such integration does not require lot of efforts to apply. From this discussion I see the “Codeception” is very popular. Who knows may be next year every PHP developer will use it, saying “anything else is a junk”.

I have nothing against its usage. But I do not like the idea to make it BDD standard for Yii at least at present time.

To clarify the question “Usage Codeception in Yii” I need to understand what exactly code changes need to be applied for it.

[Ragazzo]

Yep, you can look in codeception tests and see that they can be run through "in-browser" (acceptance) and without browser(fucntional tests (PhpBrowser module in codeception, using curl and mink), also they even can run without server through symfony browser-kit). For example simple codeception cept (cept are simply flat files, not cest, cest are classes like in phpunit+selenium) for Yii1 base app:

$I = new TestGuy($scenario);
$I->wantTo('Test index page');
$I->amOnPage('/index.php');
$I->see('My Web Application','#header #logo');
$I->click('Login');
$I->see('Login','h1');
$I->see('Username');
$I->amGoingTo('login in the test app');
$I->fillField('#LoginForm_username','demo');
$I->fillField('#LoginForm_password','demo');
$I->click('#login-form input[type="submit"]');
$I->seeLink('Logout (demo)');
$I->click('Logout (demo)');
$I->seeLink('Login');

easy to read for developer, also prints good formed scenarios that are easy to read also. So, if you will have some time just try codeception for functional (web) and acceptance(web) tests. P.S. also some users have problems with current Yii1 integration, unfortunate i am not so good developer) so you can use PhpBrowser it is also as fast as symfony-browser kit almost.

Controllers and models (see the issue title) should be testable with unit tests!

yes, tottally agree, thats why we do not discussing this section.

P.S. well there are a lot of my posts here on topic so i think i need to get out myself for some time from here))

[Ragazzo]

To clarify the question “Usage Codeception in Yii” I need to understand what exactly code changes need to be applied for it.

well, not so much changes just implementing driver for symfony browser-kit, and maybe some stubs to avoid headers flush and session_start calls (this needed only for browser-kit). As i've writed above i will provide some common pitfails that were investigated when i was developing Yii1 module.

[jacmoe]

@klimov-paul Don't you forget that with Codeception you can use all of the different functional testing tools? At least that's how I understand it. That sounds really flexible. You can't really compare Codeception with Behat or Selenium.. That would be comparing apples and carrots.

[adamaltman]

We introduced Codeception on a large project, where we only had mild PHPUnit coverage. The PHPUnit tests run, without modifications, with Codeception. We adopted Codeception in trying to solve some problems:

1. Unit testing alone was not catching some very obvious bugs.
2. Developers were not writing exhaustive unit tests. The unit tests are somewhat difficult to read, especially for a non-developer.
3. Product managers were not communicating acceptance criteria in a specific enough manner to developers.
4. The product itself was not the easiest to test manually (one piece of it is a REST api).

Within 1 week of pressing the adoption, the team wrote a series of api tests that were more or less easy enough for product managers to write (the learning curve is very short). We hit some hurdles with the integration (for example, our REST Api uses http error response codes on errors, and out of the box we couldn't test our error cases without extending and overriding some functionality of the integration). We still don't have coverage of all possible scenarios, but the team is happy knowing that we cover the scenarios that are obvious, and the scenarios that our customers use most frequently.

We also use the functional tests on our frontend. This can be more challenging, at times, and @Ragazzo has seen that we've had to extend the integration with Yii 1.1.x in order to get certain dom crawling type operations to work as expected.

[Ragazzo]

we've had to extend the integration with Yii 1.1.x in order to get certain dom crawling type operations to work as expected.

yes, confirm. @adamaltman but maybe it is a good way of course to stay on phpunit tests for unit-tests and use codeception only for functional/acceptance or api, etc?

P.S. i've finished my article, it was decided to split it into 3 parts, i think first will be available today and other in next few days, so i will also post that on yii-en forum codeception topic and hint link here.