search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.9k stars 2.65k forks source link

Continuous Integration server improvements #1470

Closed mattab closed 11 years ago

mattab commented 14 years ago

High priority/ Super useful projects

Improvement ideas

Created as a followup / summary of the 2 year old ticket #1470

More information about piwik QA: http://piwik.org/qa/

tsteur commented 13 years ago

Attachment: install.sh

mattab commented 14 years ago

Other feature ideas

Note: I read on PHP sonar "PHP at least 5.2.6 is required for this procedure and the plugins to work smoothly". It should probably only run in one build anyway (more recent php version).

I would say that following features are important to have to make Hudson very useful:

robocoder commented 14 years ago

Hudson doesn't archive artifacts for failed builds. This doesn't appear to be configurable.

I have scripted a partial workaround, copying the webtest results to http://qa.piwik.org/webtest/ and the Piwik documentation to http://qa.piwik.org/phpdocs/ (if generated).

robocoder commented 14 years ago

Build results are now emailed to the piwik-builds mailing list.

robocoder commented 14 years ago

To test release update:

Set up build server to:

Add web test:

robocoder commented 14 years ago

I created a master job called "CI" which does a single "svn up". This then triggers two jobs, "Docs" (phpdoc generation) and "Piwik" (unit tests, web tests, and schemaspy analysis), which can run in parallel.

The SchemaSpy analysis can now be found at http://qa.piwik.org/schema/

Note: since the integration unit tests are now run 3 times (non-webtest pdo_mysql, non-webtest mysqli, and webtest pdo_mysql), the "processed folder" is renamed between runs. If you browse to http://qa.piwik.org:8080/hudson/job/Piwik/ws/build/tests/integration/ you'll see the folder contains:

robocoder commented 14 years ago

Ok, default error_reporting is now E_ALL | E_STRICT. all_tests.php is now failing with/without the webtest wrapper on php 5.2.14.

mattab commented 14 years ago

Anthon, unit tests are failing now, it seems the tracking is not working anymore.

Relevant part of http://qa.piwik.org/webtest/002_UnitTests/001_response_invoke.xml is:

<span class="fail">Fail</span>: ../tests/integration/Main.test.php -&gt; Test_Piwik_Integration_Main -&gt; test_noVisit -&gt;  at [/home/www/data/root/hudson.private/jobs/Piwik/workspace/build/tests/integration/Integration.php line 159]<br />

Expected GIF beacon, got: <br/>

is there a way to access error logs of the build? would be great to have them viewable online (or inside the zip artifacts).

I expect that something is wrong there. AT the same time, it is on display_errors On so we should see the error in the response, but it seems tracker is just returning blank response? any idea or anything in error logs?

robocoder commented 14 years ago

Unit tests are finally green. (Setting allow_url_fopen=On solves the PiwikTracker problem.)

php-cgi errors are now logged to /var/log/jetty/error.log

mattab commented 14 years ago

Great job Anthon! it's all green now :) http://qa.piwik.org:8080/hudson/

robocoder commented 14 years ago

Updated webtest to R_1807 and hudson to 1.369.

robocoder commented 13 years ago

Rolled comment:2 and comment:6 ideas into main description. (LDAP login for Hudson isn't doable at this time per matt's email.)

robocoder commented 13 years ago

Updated server to hudson 1.388, and switched to a single executor. Two executors allowed doc generation and unit testing in parallel. But having 2 also sometimes resulting in double builds, and false build failures.

robocoder commented 13 years ago

(In [3433]) refs #1470 - add nightly build script; tweak the webtest build script

robocoder commented 13 years ago

(In [3434]) refs #1470 - build environment changes

mattab commented 13 years ago

A good article pointing out Code coverage report in Hudson (via phpunit again, should we migrate...) and : http://techportal.ibuildings.com/2010/09/20/building-a-continuous-integration-server-for-php-with-hudson/

Also interesting: PHP Mess detector and PHP Copy Paste.

mattab commented 13 years ago

We have to install the Chuck Norris plugin

halfdan commented 13 years ago

+1 for PHP Unit, simple test really bugs me sometimes.

For the locale, we should check a locale other than german/english. Maybe some chinese/japanese/cyrillic.

robocoder commented 13 years ago

Matt: We're now running Jenkins. For the Hudson Trac plugin, please update the URL and re-enable it. Thanks.

mattab commented 13 years ago

done.

Looking at jenkins-php.org I have the feeling we will have to migrate to PHPUnit to benefit all the cool tools around PHP QA in Jenkins. Not sure how easy it will be to change the testing library, hopefuly not too difficult.

robocoder commented 13 years ago

(In [3889]) refs #1470 - add WebTests for piwik.js (tracking-related tests are disabled because HtmlUnit behaves differently from native browser)

robocoder commented 13 years ago

(In [3891]) refs #1470 - guard against a false positive, e.g., 0 tests of 0 passed, 0 failed.

robocoder commented 13 years ago

(In [4109]) refs #1470 - test piwik.js against third-party javascript frameworks (dojo, extjs, jquery, prototype, mootools)

robocoder commented 13 years ago

(In [4110]) refs #1470

robocoder commented 13 years ago

(In [4111]) refs #1470

robocoder commented 13 years ago

Tracking requests are disabled in WebTest because the HtmlUnit wrapper loads the href on simulated clicks into the current response instead of the designated iframe (WT-428); setting evt.returnValue to false or calling evt.preventDefault() doesn't work as intended, so can't be used as a workaround. Also, GET requests (fetching the web bug/beacon/pixel) are silently ignored (by design).

mattab commented 13 years ago

It would be nice that the build runs faster, for example by only running unit tests first, then another build would run the webtests (which seem to take few minutes).

It is sometimes useful to have the unit tests feedback after a few seconds than the current 7minutes build time, especially because running the full tests on my box takes 3min and I often just run one or two and rely on Jenkins for the full run.

robocoder commented 13 years ago

Good point. I'll break this up into separate jobs, and only trigger the webtests if the unit tests pass.

robocoder commented 13 years ago

(In [4286]) refs #1470 - fail build early if unit tests fail

robocoder commented 13 years ago

Reminder to self of recent changes on dev6 that I have to update in the slack files:

robocoder commented 13 years ago

jsphpdoc - does some text transformations and feeds the output to phpDocumentor; since it doesn't parse the JS, it doesn't recognize some of the syntactic sugar to define methods. The result is missing the public methods, but includes many private methods.

jsdoc-toolkit - uses rhino/java; despite failing miserably (initial output has only three functions: cvt_hex, piwik_log, and piwik_track), it's probably the more promising of the two given its feature set (and usage by other projects). It appears to we have to add more tags as hints...

robocoder commented 13 years ago

(In [4380]) refs #1470 - add jsdoc-toolkit

robocoder commented 13 years ago

ok, Piwik Mobile docs are online at http://qa.piwik.org:8080/jsdocs/

I've updated the QA page, too.

robocoder commented 13 years ago

PHP 5.4 dropped the sqlite extension -- required by tests/javascript/.

mattab commented 13 years ago
sgiehl commented 13 years ago

I allready migrated most core and plugin simpletests to phpunit on my local copy. When I got some time I'll have a close look at the integration tests. don't know wether they are easy to migrate or not.

mattab commented 13 years ago

Great to hear about your tests with Phpunit!!! exciting.

Migration tests should be the same to migrate as other tests, they don't use any special logic.

sgiehl commented 13 years ago

There is a small problem with the integration tests and some other tests. As simpletests are called in a browser there is a local URL to send api-requests to. phpunit is executed on command line and so there is no url. guess we need to put that to a config file in order to have the tests run correctly with phpunit.

robocoder commented 13 years ago

Right. Jenkins sets the following environment vars:

export DISPLAY=:0.0
export REMOTE_ADDR=127.0.0.1
export HTTP_HOST=localhost:8080
export REQUEST_URI=/path/to/all_tests.php
robocoder commented 13 years ago

Upgraded qa.piwik.org to WebTest R_1812 last nite. No regressions. http://qa.piwik.org:8080/jenkins/job/Piwik/2372/

mattab commented 12 years ago

Work for Simpletest to PHPUnit migration is done in: #3227 Will be a great improvement to have code coverage metrics, can't wait to see how we are doing...

EDIT: see also this presentation about Jenkins and PHP http://www.slideshare.net/sebastian_bergmann/integrate-your-php-project-with-jenkins-oscon-2012 by PHPUnit dev

mattab commented 12 years ago

Many times our sql update code wasn't failure-friendly and would fail when re-executed as it left the DB in a half-updated state.

Having the following test would ensure that all future update scripts written are working in a failure scenario.

A new idea for a webtest:

robocoder commented 12 years ago

Dropping Sonar from the todo list.

mattab commented 11 years ago

See ticket for Webtests request: #2935

This ticket is mostly implemented after 2 years! See #3606 for the follow up ticket listing all ideas and improvements to our continuous integration server and processes. Good times!