Application vs. System Testing

[jamierumbelow]

This isn't so much of an issue but more of an open question as to where unit testing support is heading for CI3.0.

It's awesome that unit tests are now a part of CI core, and it's great that the branch has been merged in and everything's starting to look good and official. I'm excited.

As you'll all know, currently the tests are in /tests. While this is fine, that folder seems to only facilitate system tests. What about application tests? It would make sense to move the /tests directory into /system/tests and have an /application/tests directory with a sample test inside it.

Is there a framework in place for application tests? There'll need to be mocks setup, methods to override core bits of CI functionality etc.

Is there anything in there that I'm missing to allow for this yet? If not, want me to look into it?

[toopay]

and if we remove all comments block from the system/ code, tests/ will already have a third of it's size.

Its a consequence of having unit-testing. A series of test agains API from one class, will be bigger that the class/file itself. Something like swiftmailer going crazy on their test-suite size.

If not anything else - we need to focus more on CodeIgniter's quality and new functionalities.

There is a direct relation, between having test-suite and all of these holla-mumbo-jumbo code-coverage, with code quality. It is also ensure any new functionalities not break the current one.

[philsturgeon]

Agreed. Before we can even consider large changes we need to have existing code well tested with high code coverage, so we can see it break and make sure it's fixed.

Emailing from my iPhone 4S like a BOSS!

On 7 May 2012, at 07:26, Taufan Adityareply@reply.github.com wrote:

and if we remove all comments block from the system/ code, tests/ will already have a third of it's size.

Its a consequence of having unit-testing. A series of test agains API from one class, will be bigger that the class/file itself. Something like swiftmailer going crazy on their test-suite size.

If not anything else - we need to focus more on CodeIgniter's quality and new functionalities.

There is a direct relation, between having test-suite and all of these holla-mumbo-jumbo code-coverage, with code quality. It is also ensure any new functionalities not break the current one.

---

Reply to this email directly or view it on GitHub: https://github.com/EllisLab/CodeIgniter/issues/1315#issuecomment-5548837

[yoosuf]

sorry about Out of the chapter

@philsturgeon i like your email signature

[ttessin]

Just tuning into this thread. I have been using PHPUnit/Selenium now for quite a while in our CI based web application. We started with Selenese (xml) and converted it to PHPUnit. We reduced test time from 5+ hours to less than 45 minutes using parallel threads. I have since started writing plain PHPUnit tests for our 'bolt on' libraries and included pear libraries, and just started playing with PHPUnit tests for controllers/models/libraries. My directory structure has /system, /application, and /tests all at the same level. /tests has the following subdirs: selenium (all PHPUnit/Selenium tests), phpunit (vanilla PHPUnit unit tests) and api (home grown command line api tests). phpunit is further subdivided into models, controllers, and libraries and ciunit(fooStack). We have multiple phpunit.xml type files (some of them are auto-generated in the build to do parallel test runs).

We can work with system/tests and application/tests if that is the decision. However having our application test code in a /tests directory allows me to zip it to a selenium driver server and farm out the test threads. /tests/selenium can run independently of the CI code base so the selenium driver server doesn't need to have the rest of the app code checkout.

[mastacheata]

This discussion obviously stopped about half a year ago. Did you come to an agreement of any kind behind the scenes or did this just get lost over the summer?

Nevertheless I'd like to offer my opinion on the Application Testing stuff: You all discussed about whether and how many subfolders to create in application/tests/... I would appreciate an all empty application/tests folder or at most a very basic PHPUnit Testcase covering the default controller. There should not be any hierarchy like in the system part.

This would lead to too many different possiblities. People can't even agree where controller starts and model ends or how many intelligence may sit in the view. (Is a loop inside a view already too much, should validation be performed in controller or in model etc.)

Even though some of the professional users or even dedicated amateurs would really like to see Test Coverage in all possible areas I doubt that you will convince the majority to use it. Keep in mind everyone is complaining about missing or insufficient documentation since the beginning of computers and even before that.

CI should not force users to do tests, it should make it easy enough for people to want to do testing. This is also not about which Test Framework is possibly better. PHPUnit was already chosen for Unit Testing the CI-Framework itself, so we should stick to that and offer the facilities to easily extend this to the Application. How to extend that on the application should then be explained to a greater extent in the User Guide.

Another matter was that the current Test Coverage is mostly achieved with the help of the Travis Continuous Integration System. It is rather safe to assume that this is not available to most users. I wouldn't mind the System Test Coverage to stick to that, but it must definitely be optional for Application Testing.

[alexbilbie]

CI won't force users to do tests and it's never been suggested it will.

As Phil has said before, we need to bolster the existing unit tests and add tests for classes that haven't yet been covered so that the core is water tight before we can go any further on this

[mastacheata]

Sorry, maybe I chose the wrong words there. What I meant was not so much that this was ever thought to enforce tests on the user. I meant it more like it's a bad idea to have a too tight path for testing the application part. (And that was at least discussed here even though it was mostly discarded)

I can fully understand that you want to focus on getting the base of codeigniter covered by tests first before deciding on any final way for application testing. Nevertheless collecting ideas and opinions should still be fine even while the core stuff is far from finished.

[AkenRoberts]

From what I gather here (and agree with), zero unit testing will be included with the stable production release of CodeIgniter - system, application, or otherwise. Unit testing is being used to aid in development of 3.0, not as a released feature. Application unit testing should be encouraged via the user guide and/or a Spark. Anyone interested in contributing to the guide or a spark should do just that.

[alexbilbie]

@cryode to clarify there will be system unit tests in CI3, there might not be a way of doing application tests yet

[AkenRoberts]

Won't that force users to make sure a particular testing suite is installed / available?

[alexbilbie]

Does it actually matter if /system is tested with PHPUnit?

When it comes to /application testing I imagine there will be a number of hooks into CI so if you want to test in Behat you can or you can stick with PHPUnit

[AkenRoberts]

I don't really know if it matters or not; I have limited experience with unit testing. I'm legitimately curious.

[igbanam]

Why all this fuss about application-level testing? If need be, CI has somewhat embraced Composer. Add this to your application and test using Codeception. The CI tests are CI's tests; completely orthogonal to an app's test (a point over-hammered). But this begs the question though… what does CodeIgniter mean by "Application Tests" in the test/ folder?

[ivantcholakov]

@jamierumbelow

"One of the benefits of introducing unit tests to the system is that we can also introduce unit testing to the ecosystem, that is to say, start to try to grow a culture of writing tests in our applications. Only a fool would disagree that programatic testing is a good thing to do."

I am that fool to disagree.

Jamie, some time ago I forked your project codeigniter-base-model. I liked it, of course. But I sensed your approach "Where are your tests?", it kills motivation. So, I worked on the forked project and use it for production purpose. If you make a comparison, you will see, that your original project is lagging behind.

People ask new features, and they do not have them, because you decided to stick with TDD methodology. You even removed a feature (multiple database connections), just because you was not able to test it.

I bet on progress, not on tests. -1 from me.

I agree with @derekjones that TDD is a specific methodology, and the framework should not promote it.

P.S.: A humble advice: Author of a piece of software != author of the corresponding tests.

[stgeneral]

Here is a way how I've solved the problem of tests in existing CodeIgniter app https://github.com/stgeneral/CodeIgniter-TestSuite. Solution was performed on CodeIgniter 3.0 (master develop branch snapshot from Oct-Nov 2013).

[narfbg]

master is not 3.0.

[stgeneral]

Sorry, @narfbg. I've missed that default branch is develop. Fixed my previous comment. By the way, the ideas used in my suite very likely will work for 2.1 too.

[drew-hoffman]

Anything that promotes more unit testing is good in my book.

I'm a fan of Application/tests and System/tests, if anyone cares.

[bgeesaman]

I have just been charged with refactoring a good chunk of a CI 2.2 codebase with ~47K loc find ./applicaiton -name '*.php' | grep -v "third_party" | xargs wc -l and zero tests. I came across this thread in looking for the best method for integrating a unit testing framework so I can write tests to cover certain areas before I start refactoring safely. The built-in CI unit testing library is insufficient, and despite their best efforts, community implementations are rife with self-admitted "hacks" to get things working.

I find it extremely suspicious that the "core" is unit tested with PHPUnit, but extending that structure to the "application" space is considered undesirable and would go against the core tenets of CI to not "force" you into any methodology. Picking a primary testing framework and making it usable for the application side in NO WAY means I have to actually write test files or even run them. It doesn't force you into TDD or test-after-the-fact. But what it does do is make sure that there is a common and supportable way to hook into the framework to run tests that other testing frameworks can leverage if one liked those instead. As it is now, that interface doesn't exist. Any testing framework I shoe-horn in can't be guaranteed any sort of compatibility going forward. That is why I believe the decision to refrain from making the built-in CI test framework full featured (fixtures, mocks, environment handling, web drivers, etc) or even providing a somewhat legitimate integration path for external testing frameworks is so short-sighted.

This example is one of the main reasons frameworks with the tenets of "remove any friction for the developer to create and refactor quality code" are thriving and frameworks that haven't adopted critical features into their core approach are on life support. I consider having a full-featured testing framework as a core feature to be "removing friction". I'm spending days/weeks trying to bolt testing on to this codebase after the fact, but all I can think of is "What would it take to rewrite this all in ____ framework? It's probably less effort and a better long-term decision."

+1 for picking one, integrating it, documenting its usage, and treating it like a first-class citizen. application/tests and system/tests makes sense as an approach for clean separation.

[kenjis]

I wrote a bridge tool called ci-phpunit-test for CodeIgniter 3.0 application testing. http://kenjis.github.io/ci-phpunit-test/

And I sum up request list for CI 3.x, just for your information. https://github.com/kenjis/ci-phpunit-test/blob/master/docs/RequestsToCodeIgniter.md

+1 for system/tests and application/tests

[jagandecapri]

@kenjis I am using the bridge tool that you wrote. Pretty neat. It will be nice if it is included in CI core and further improved. :+1:

[kenjis]

@jagandecapri Thanks!

[ghost]

@kenjis +1. Works fine, well documented and add a good tool for developers

[feryardiant]

+1 for @kenjis, actually I'm not using your bridge but I do aggreed that Codeigniter should be testable.

Writing tests for CodeIgniter 3.0 application has troublesome. Because CodeIgniter has some code which is untestable or difficult to test.

I'm using this one and had same problem.

[kenjis]

@kfern-github Thanks!

@feryardiant all the problems stated in the request list have been solved (or there is workaround) in my ci-phpunit-test.

[feryardiant]

@kenjis great!