Reduce number of jobs in the Travis CI piwik/piwik build

[mattab]

Currently each commit on piwik/piwik repository will trigger a new build on travis CI with 20 different jobs. The goal of this issue is to go down to hopefully 10 jobs instead!

Context

• Our build configuration has 20 jobs which is rather extreme. we run tests for Unit+Integration+System tests for PHP 5.3, 5.4, 5.5 and 5.6 plus an extra run for Code coverage #4302 and an extra one for MYSQLI tests (default uses PDO).
  • Running suites in separate jobs has advantage to get feedback about which test suite failed.
  • but it means we create for each commit 3 test suites * 6 = 18 jobs

Proposed solution Instead we could adopt a smarter strategy... :8ball: which consists of:

• First we run a job for each test suite in PHP 5.3 in order: one job for system, one job for Core and one for Integration -> it gives devs quick useful feedback.
• Then we run one job for each PHP version. In this job we run Unit tests, then Integration tests, then System tests. We save two jobs per PHP version.
  • This would save ( 5.4 + 5.5 + 5.6 ) * 2 jobs = 6 jobs
• We could run MYSQLI tests in one job -> saves 2 jobs
• maybe we also could run Code coverage in one job -> save 2 jobs

As a result we should get much faster build time and build feedback #6412

[sgiehl]

As long as the tests are taking so long to run, I don't see any option to combine the codecoverage builds. Tried that while implementing, but always reached the travis job time limit, and the job was canceled.

[tsteur]

I shrinked it down to 9 jobs + 3 for coverage so far. Not sure if it works yet.

Re coverage: I think it makes sense to remove at least the SystemTests code coverage job as the coverage for this is basically useless. Even for integration tests it is kinda useless but as the job doesn't take that long it is a nice to have. Maybe we can remove the one for SystemTests (which takes > 50min) and run only Unit and Integration instead?

[mnapoli]

:+1: for removing coverage for system tests.

And even :+1: for removing it for integration tests but that's less critical.

[mnapoli]

Anyway the coverage for system tests is killed by Travis because it takes more than 50 minutes: https://travis-ci.org/piwik/piwik/jobs/38106958

So I think we can safely remove it.

[tsteur]

FYI: Not running PHP 5.3 tests first as many tests are skipped for PHP 5.3 (which we should not do BTW but that's another topic). Therefore running 5.4 first and the suites separately for 5.4

[diosmosis]

which we should not do BTW but that's another topic

There are A LOT of skipped tests. Not just for 5.3, I wanted to re-enable them (created a branch for it), but couldn't find the time to finish it.

[tsteur]

@sgiehl now that we run only coverage for Unit and Integration tests can we maybe combine it in one job but still get two different code coverages for them?

@diosmosis nice! yeah we should really fix the remaining ones ASAP and it should be no longer done at all in any future test case

[sgiehl]

@tsteur I don't think that would work as coveralls would combine the both results before sending them.

[tsteur]

alright. I'm not so deep in this topic therefore maybe stupid question: Can't we create the coverage with the 5.4 Unit and Integration test maybe? So that we can get rid of both completely. I reckon it is not done this way as they take much longer with coverage?

[sgiehl]

Sure we could. I separated them as, as you supposed, they take much longer. And failures caused by code coverage, such as timeouts, are ignored that way.

[tsteur]

Just had a look if we can speed up coverage generation but haven't found anything useful. So maybe we leave those two jobs there for now as 20 minutes for integration tests take too long.

While looking for this I noticed disabling xdebug reduces the test time on my instance by about 30% (just normal that it will be faster). For the default tests we won't need xdebug so will check if this is doable. I saw the performance improvement after disabling the extension, not after calling xdebug_disable().

[tsteur]

Tests are now about 10-30% faster I think as xdebug is disabled. AllTests from > 22 minutes down to 15minutes. UnitTests from 30 seconds to 23, IntegrationTests from 6:30 to 4:30 or so. Of course travis speed always varies but I can see this on my local instance as well. Should be at least 10% faster. See https://travis-ci.org/piwik/piwik/builds/38222406

[mnapoli]

Merged #6459: we now have 9 jobs per build instead of 11. Tests run only on 5.3.3 and 5.6.

[tsteur]

Perfect!

[mattab]

A so very welcome change! :+1:

[tsteur]

I know it is already closed but don't want to create a new issue for a random thought. Maybe we could remove one more job PHP 5.6 TEST_SUITE=UnitTests MYSQL_ADAPTER=PDO_MYSQL and move the PHP 5.6 Unit Test with Coverage upwards where the previous Unit Test was executed before and do not allow failures for this one. They both test the same.

I think the normal Unit Test takes 2 minutes on Travis whereas the other one with coverage takes about 5 minutes which is not a big difference for me. Unit Test should not fail so often and anyone who needs a fast test result can execute them locally within 10-30 seconds.

[mnapoli]

@tsteur Code coverage builds are in the process of being moved to Scrutinizer, so they should disappear from Travis anyway (that'll be 2 less builds): #6556