Feature comparison to PHPUnit?

[milesj]

I'm looking to replace PHPUnit with HackUnit on Titon: https://github.com/titon/framework

But before I do that, I would like to know what features in PHPUnit are in HackUnit, or what's missing? Is this project stable enough to use?

This project looks promising. I would love to use it at some point.

We're also looking for help/contributions on Titon if you're interested.

[kilahm]

Thanks for your interest @milesj! HackUnit is just barely getting out of proof-of-concept stage, so definitely not "production ready". I'm not sure the goal is to copy all of PHPUnit's functionality, so that comparison may not be quite useful.

I will point you at TestCase.php to see how assertion objects are instantiated. CallableExpectation.php and Expectation.php show what can be done with a particular assertion object.

I am in the middle of a re-write of some of the core functionality, including discovery of test classes and which methods of a test class are actually tests, how output is generated, and what types of assertions can be made. I'm not sure how much time this will take as I have very little time to be able to devote to this project (though I really want to!). I plan on pushing a 0.4-dev branch to the repo so people can preview what I'm working on.

Part of this re-write will also include better documentation, likely hosted on readthedocs.org. I can keep you posted as I expand this project.

[milesj]

I'll hold off on integration until you finish that re-write and get it to a stable point. I'd also be willing to contribute at that point, so be sure to keep me posted! Thanks again.

[steve-rodrigue]

I would really like to contribute to this project. @kilahm do you have a list of features that you are currently working on for 0.4-dev and a list of features we must have compared to what xunit/phpunit currently has?

[kilahm]

Thanks for your interest @steve-rodrigue! Unfortunately 100% of my coding time has been dedicated to work, but I have big plans for this project. I just haven't found an afternoon available to devote to this project in the last two months. (Work is attempting to release version 2.0 about two weeks ago, and I'm currently planning my wedding which will happen on Aug. 1)

I'm envisioning a complete rewrite, using the hackpack/hack-class-scanner package as the test files loader, kilahm/clio to handle command line arguments and feedback to the user, and using reflection to read user defined attributes (http://docs.hhvm.com/manual/en/hack.attributes.php) to mark methods as test/setup/teardown. You can look at my attribute based Hack router project (https://github.com/kilahm/attribute-router), or my IOC container project (https://github.com/kilahm/IOCFactoryContainer) for how I envision using the class scanner to load files based on user attributes.

I'm quite confused by the current loader/instantiator and would like to completely rewrite the whole entry point to the app. I'm envisioning the hackunit executable loading the composer autoloader, instantiating a Clio instance, passing it to a config class static method which will configure it and read the appropriate command line arguments. The result would be a config instance that would be injected, along with the clio instance, into an app object, then a run() method would be called on the app object. The whole thing could also use my IOCFactoryContainer project for dependency injection.

The current implementation uses events to trigger user feedback, which I keep going back and forth on actually using. The problem with a generic event based system in Hack is that the callbacks cannot be statically typed in a generic way. Although I also have an idea to get around that: An emitter trait that holds a list of callbacks to trigger on specific events that keeps all of the registrations methods protected/private. Then the class using the emitter trait would have to expose listener registration methods with a specific callback signature that then call the trait method(s). This would mean there is no need for a listener class/trait. All listeners are just callables that adhere to the correct function signature.

If we were to go full event based triggering of user feedback (which could be cool) then we should get rid of all of the assertion exceptions as carriers of user feedback. Rather there would be one assertion exception that would halt the execution of a particular test. It would also probably mean creating an Assert class similar to xUnit that would handle all of the actual checking of values and trigger the pass/fail/skip events, while the test runner class could also trigger the skip event based on a user defined attribute.

I just realized that this brain-dump didn't directly address your question of a list of features, esp. compared to xUnit or phpUnit. Honestly I'm not worried about comparing this project to those. I think the main driving goal of this project from its creator @brianium was a pure Hacklang implementation of the xUnit concept. In that spirit, we are not depending on any PHP libraries, and if we find we would like some functionality in a PHP library, we would like to re-imagine it using pure Hack.

Given all above, if you have ideas for what this project could be (respecting the pure Hack goal) I would love to hear them!

[kilahm]

I was able to work on this project over the weekend! @steve-rodrigue if you are still interested, please look over the code I pushed to the 0.4 branch, as well as the new README. Does my approach make sense? Do you like the use of user attributes? How about the "event" system I used?

[steve-rodrigue]

Sorry I didn't get back to you earlier. I will have a look at your code during the week end.

I had to support mock objects for my project, and I needed an API that looks like the one of PHPUnit to easily port code from PHP to Hack lang. For these reasons, I decided to create my own project: https://github.com/irestful/hack-unit

I should have some time during the week end to finish the tasks written in the README to release it.

I know I might have a different vision than yours for this project, but if you think we could work together, let me know. I just need to have the API similar to the one of PHPUnit in order to make my code port easier to do. I am sure a lot of people will have similar needs, don't you think?

Cheers! :)

[kilahm]

I looked briefly at your support for mock objects, and I don't see how that approach could work in strict mode. Namely the use eval and dynamic class instantiation (new $var()) in AbstractHackUnitTestApplication::getMock(). These two constructs go against the philosophy of static typing, i.e., explicitly declare the types of the objects you are instantiating.

I would argue that programmatic definition of classes itself goes against the philosophy of static typing. Thus even having the concept of dynamic mocks makes the static type checker less useful since the type checker has no way of knowing that the mock you created implements the interface or extends the abstract class you have listed as a dependency on the subject under test.

It seems to me that test doubles should be explicitly defined in their own files, and injected into the subject under test. That way the static type checker can analyze their code and ensure all of the type requirements are consistent.

I believe supporting the type checker as much as possible is good because if the type checker can know all of the types of all the objects in the project, no unit tests need to be written about checking the type of anything.

Like I said at the top, I only briefly glanced at your code, so I haven't had time to really think about it or grok the structure you've come up with. But I think we do have different visions for a Hacklang xUnit framework. I'm not worried about compatibility with phpUnit at all. Honestly, I've only used phpUnit very briefly, so I'm not all that familiar with its features and conventions. I guess I'm going with what makes sense to me in the context of a dynamically run statically typed language.

[belka-ew]

@kilahm Hi, I'm the next person interested in a testing framework in Hacklang :) I tried to install 0.4-dev and the type checker complains then about a lot of errors because ClassLoader and CLI ship 0.3.x Tests, that aren't converted. Without these tests everything looks well. What is the current status of your work on HackUnit? Can I help in any way?

[belka-ew]

@kilahm I looked recently into the Unit Testing in D language. In D you put the tests not into a separate class but into the tested class: http://dlang.org/spec/unittest.html. Using attributes you introduced it would be possible do the same in Hacklang. It would make testing of private and protected members and some mocking cases very easy. And some people could like this idea to keep the code and the tests together since Tests/ has mostly the same structure as the source code. And who doesn't like it can still put their tests else where. "TestSuite" has to be removed then and all methods have "Test" attribute set, would be considered test methods. What do you think about this approach?

[kilahm]

I had that same thought though I never actually tried implementing it. I think it's one of the biggest advantages of using the annotations for test methods.

There would need to be some thought about the <> annotation though. Would it still be needed? The reason I included it was to cut down on the load time by restricting classes being scanned. I'm not sure that matters now that all files will be scanned.

[belka-ew]

I would remove "TestSuite". I would remove it though if the tests would be in a separate directory as well :), it is kind of overkill for me to annotate everything.. too many attributes. Just my opinion.

[kilahm]

I'm going to close this issue because the conversation has strayed from a comparison between PHPUnit and HackUnit.