testing

[ianstormtaylor]

@segmentio/atwix how do you guys normally setup testing for Magento plugins? We'd like to get tests setup that can be run easily from the command line so that whenever we merge in fixes we can be sure that we haven't broken the existing functionality.

When I was building out a Magento plugin a while back I copied a technique from WordPress plugins where I just added a test/ folder with a complete Magento install that symlinked to the plugin code itself. Then with MAMP I would just load it up and test things. Does that make sense? or seem like a bad way to do it?

Would love to hear what you recommend.

[yaro-atwix]

When we use unit tests, we usually put a directory with tests alongside with a extension's code in the separate GIT branch which is not being merged into the master branch. It allows to have all tests in the repository and hide the tests from the release branches (master) at the same time. As for functional testing, we usually hold all our tests separately from the extension's code (it might be a separate repository) since functional tests don't connected to the extension's code directly at the most cases.

[ianstormtaylor]

Question: when we put it in the Connect store, is it most likely that 99% of people will install it through Connect and not by downloading the code directly?

If so, I think it might be good to keep simplicity on our end down by actually just including tests in the master branch, so that we can easily run make test or something similar and make sure everything looks good. If we wanted, we could have a folder structure like:

• /plugin - the actual code for the plugin.
• /dist - ZIP files for the releases, for example analytics-magento-1.0.2.zip.
• /test - all of the code required to test, both unit and functional.

From our end we want to get the testing process down to a single make command so that it's very simple for us to release new versions.

[yaro-atwix]

As for testing using simple command: you are always able to do it, no matter where is the tests are placed. Just use use symlinks, bash script, relative paths etc. So, a separate directory for test is a good solution. As for dists directory, I'm not sure it's necessary in the repository. I don't think that there is a big count of users who need an outdated version of the extension (in case if you want to keep all releases there). Well then, you are always able to get an older version using Git.

[ianstormtaylor]

I think in the end here we are going to want to go with keeping the tests in this same directory. Basically we'd move to a structure where the plugin isn't top-level itself, but nested:

/analytics-magento
  /plugin
  /test

Let me know how we can proceed there @yaro-atwix or what you need from my end. I think we'd like to get both unit tests for the lower-level things, as well as some end-to-end tests with PhantomJS or Nightmare so that we can make sure the user flow is working.

The goal is to be able to have to make commands:

make server    # runs the testing sure
make test      # runs the tests

[yaro-atwix]

We are using CasperJS for functional testing. It's a 'framework' for PhantomJS which make tests creation more fast. Is it ok for you? We need about 7.5 hours to complete our functional tests and write the unit tests.

[ianstormtaylor]

Yup that sounds perfect to me!

[yaro-atwix]

Hello Ian. I'm on a business trip at the moment. Will be able to get back to the work after May 18. Plan to finish with the tests in a week after this date. Is it ok for you?

[ianstormtaylor]

Hey, yup that sounds good. Just let me know as soon ads they're ready and I can check them out. We're hoping to do a press launch with this plugin with a few others soon

[yaro-atwix]

Have a small question regrading the functional tests. There are two ways to make the tests. The first one will check necessary code presence on the corresponding pages. The second will check code presence and the data inside of the code (i.e user name, order total etc). But the second one works only with a clean Magento installation and native sample data. At the same time the first one is more universal. However, the universality is limited. For example, 3rd party checkout extensions need to be covered with tests separately. I would recommend the second.

[ianstormtaylor]

Nice, the second one sounds good to me.

Let's include a simple Magento installation inside this repo in the /test folder, and then make a script that can fill it with the test data we need. (I think I remember seeing that there is standard test data available somewhere? we could use that.) So for a file structure we might have:

/plugin
/test
  /magento
  /data
  index.php

If that makes sense. I'd like to keep all of the necessary logic in this repository, so that we don't have to worry about the maintenance across them all.

[yaro-atwix]

It is a good idea. But what about different Magento versions? Do you plan to keep them all in the repository?

[ianstormtaylor]

What are the major ones you normally test against? or what are the major ones people still use? I hadn't thought of that, but we can be a bit cutting-edge in what we support I think? Let me know what you advise there.

[yaro-atwix]

I guess currently it will be enough to put CE1.8 and CE1.9 since these versions has some differences that might cause troubles with the extension.

We almost completed our tests. However, the new sample data has been out a week ago and it's completely differs from the previous one. We decide to ged rid of using the sample data in our tests. It will not take much time.

[yaro-atwix]

We've just finished with tests. We got rid of using the sample data, so, all tests might be used on a clean Magento installation. Also, there's a shell script called tests/test.sh for running all tests in one command. The example of usage is following:

./test.sh http://siteurl.com/

The script runs all unit tests first (the unit tests also create test product and category for functional tests) and then runs all functional tests.

You are able to see it's 'demo' here: http://screencast.com/t/YNkFMbxb8lx

All tests have been run on clean distributives of Magento CE 1.8-1.9

Please, let us know if you have any question

[yaro-atwix]

We've written the small HOWTO about installation and testing. It's in the Readme.md as you can see.

[ianstormtaylor]

Thanks!