Jenkins builds Fatal error: Allowed memory size of 2147483648 bytes exhausted

[photodude]

Steps to reproduce the issue

Look at the current Jenkins builds console output http://build.joomla.org/job/cms/6586/console

 **[exec] Time: 18.04 minutes, Memory: 1085.00Mb**

 [exec] OK, but incomplete, skipped, or risky tests!
 [exec] Tests: 5480, Assertions: 10096, Incomplete: 28, Skipped: 345.

 [exec] Generating code coverage report in Clover XML format ... done

 [exec] Generating code coverage report in HTML format ...
 [exec] Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 25849 bytes) in phar:///usr/local/bin/phpunit/php-token-stream/Token/Stream.php on line 149
 [exec] Call Stack:
 [exec]     0.0038     313448   1. {main}() /usr/local/bin/phpunit:0
 [exec]     0.0432     738272   2. PHPUnit_TextUI_Command::main() /usr/local/bin/phpunit:586
 [exec]     0.0432     738488   3. PHPUnit_TextUI_Command->run() phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132
 [exec]     4.1895   94310400   4. PHPUnit_TextUI_TestRunner->doRun() phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
 [exec]  1136.6806 1768559528   5. PHP_CodeCoverage_Report_HTML->process() phar:///usr/local/bin/phpunit/phpunit/TextUI/TestRunner.php:474
 [exec]  1136.6806 1768560128   6. PHP_CodeCoverage->getReport() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/HTML.php:110
 [exec]  1136.6806 1768560224   7. PHP_CodeCoverage_Report_Factory->create() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage.php:165
 [exec]  1136.7729 1777362904   8. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:75
 [exec]  1136.7765 1777402168   9. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:97
 [exec]  1136.7765 1777403592  10. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:97
 [exec]  1136.7765 1777405032  11. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:97
 [exec]  1136.7766 1777406504  12. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:97
 [exec]  1136.7766 1777407936  13. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:97
 [exec]  1136.7766 1777409328  14. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:97
 [exec]  1173.5394 2131196424  15. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:97
 [exec]  1173.8596 2134886392  16. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:97
 [exec]  1173.8807 2135143480  17. PHP_CodeCoverage_Report_Factory->addItems() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:97
 [exec]  1173.8808 2135144032  18. PHP_CodeCoverage_Report_Node_Directory->addFile() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Factory.php:93
 [exec]  1173.8808 2135144312  19. PHP_CodeCoverage_Report_Node_File->__construct() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Node/Directory.php:210
 [exec]  1173.8808 2135144312  20. PHP_CodeCoverage_Report_Node_File->calculateStatistics() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Node/File.php:163
 [exec]  1173.8808 2135144656  21. PHP_Token_Stream->__construct() phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage/Report/Node/File.php:397
 [exec]  1173.8809 2135162392  22. PHP_Token_Stream->scan() phar:///usr/local/bin/phpunit/php-token-stream/Token/Stream.php:152
 [exec]  1173.8809 2135162504  23. token_get_all() phar:///usr/local/bin/phpunit/php-token-stream/Token/Stream.php:195

Expected result

No Fatal error on Jenkins build

Actual result

Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 25849 bytes) in phar:///usr/local/bin/phpunit/php-token-stream/Token/Stream.php on line 149

System information (as much as possible)

Jenkins

Additional comments

none

[mbabker]

builds should be closer to 5 minutes

The Jenkins server's builds have always been historically slower than Travis because the architecture isn't as finely tuned (this is much closer to a production hosting environment than an environment built with the purpose of running a lot of test builds efficiently). So I don't think it's fair to raise an issue here about the timing difference.

The memory issues are 100% fair though.

[mbabker]

Also consider that our Travis builds are not collecting code coverage, Jenkins does with the explicit purpose of publishing reports at https://developer.joomla.org/cms-coverage/

That change alone causes a major difference in memory consumption and how long the process takes.

[photodude]

I'll take your word on the Timing being acceptable. Off the top of my head I thought the builds were less than 10 minutes before, rather than close to 20 minutes.

But let us focus on the most obvious and significant of the issues and see try to address the Fatal error in the Allowed memory. As such I've edited the issue to reflect that focus.

[rdeutz]

it is a mess, I think jenkins is not running very well these days and that is also true for the some weeks in the past. We really have to look what is the purpose of running jenkins, my guess is we can do it in a better way with less problems.

[photodude]

@rdeutz As far as I know Jenkins runs with the explicit purpose of publishing report code coverage at https://developer.joomla.org/cms-coverage/ and does nothing else.

[mbabker]

It's not doing much useful anymore for the CMS repo since we've decided to try out or use all sorts of other CI systems.

[rdeutz]

it is also merging staging into master, most of the testing work is done by travis (all sorts of === travis)

[mbabker]

Someone actually uses master for something?

/me hides

[rdeutz]

can't say, we might should asked the people who have set it up this way :-P

[csthomas]

If you need a help with jenkins I can try to help.

Notice: 1) PhpUnit is a little old at http://build.joomla.org/ 2) Memcache and memcached extensions are loaded but memcached server is down.

[mbabker]

When we set up this version of the Jenkins server the CMS was still running PHPUnit 4.1 because of the deprecations in later versions causing failures (since addressed) nor was Composer integrated. I honestly haven't cared enough to go back and change the job to use the Composer dependencies over the globally installed PHPUnit PHAR since it still works.

The intent also was to try and get as many of the "optional" dependencies running on that server for accurate code coverage reports. So the resources for things like Memcache(d), Redis, APC, and PostgreSQL are all there, whether or not all the required services are running anymore is another story.

If we're going to phase Jenkins out of the CMS workflow I don't want to invest time in configuration changes personally; it works fine for the other tasks we're using the server for and I want to keep pushing more deployment related tasks for the Joomla websites onto it so things are less reliant on me connecting into servers to run whatever tasks are needed.

If we're still going to use it for something CMS workflow related (I'd say code coverage reports myself; personally I'm not comfortable, even with encrypted variables, giving Travis access to push files onto one of our hosting servers), I've got no problem getting things updated.

[rdeutz]

todays Jenkins can do a lot more with pipelines and docker support, so for me is it more a decision what and how we will do it in the future. Travis isn't a perfect solution but it is easy to setup. Couldn't it be a interim solution to update phpUnit as a first step to get it back to stable builds.

I have access so I could do it, nothing you @mbabker must do

[mbabker]

Couldn't it be a interim solution to update phpUnit as a first step to get it back to stable builds.

Composer's installed on the server, so the testing/coverage job really just needs to be updated to run composer install then trigger phpunit and phpcs through that instead of the globally installed binaries. We probably should also tweak the job so that it rebuilds the workspace on every build that way we don't need to worry about running cleanup tasks at the start/end of each job (namely composer install --no-dev to get rid of the dev dependencies again).

And while we're on the topic of Jenkins, we probably should do something about the nightly build script which is 1000% disconnected from everything.

[mbabker]

Jenkins can do a lot more with pipelines and docker support

Not necessarily related to this task, but if we're going to use Docker we need images that can test WinCache and XCache support on the cache and session APIs. Otherwise we need to drop support for those platforms completely as they're pretty niche and right now 100% untested. XCache and XDebug can't be installed in parallel and WinCache is Windoze specific.

[yvesh]

Speaking from the experience we have made at Weblinks and the GSoC automated testing project in the last months, we are quite frustrated with Travis currently (also the experience of other developers using travis for system tests)..

Almost every 3rd run (and we do 3 different version per PR) for weblinks is failing nowadays (without any code changes). It seems some methods (valid ones) are more likely to cause issues then others and we do our best to work around these, but in general Travis became quite unstable.

Within the JavaScript tests in core we have also hiccups with Travis, so Robert set allow failure for them.. Restarting the travis job fixes it. We currently run the JavaScript tests in parallel on drone.io for the last two days and hadn't had a single break on them.

So when we want to integrate the selenium tests into core (a PR will follow soon from GSoC project), without allow failure, we also need to address this issue..

[rdeutz]

seems I killed jenkins, added a shell script to call composer install and changed the build.xml so that it calls the phpunit in libraries/vendor/bin. Also bad luck, the jenkins user can't restart jenkins. I need someone with a better idea :-)

[mbabker]

Call Rochen 😛

[rdeutz]

I don't have access to the ticket system :-(

[mbabker]

/me goes to save Robert's butt

[rdeutz]

I made sure jenkins doesn't fail anymore :-P

[rdeutz]

So now jenkins

• deletes the workspace
• makes a composer install
• run unit tests with code coverage report

I also removed phpcs because we are doing it with travis, means jenkins is now creating the code coverage reports and merging into master

[zero-24]

Merging into staging or master?

[rdeutz]

ah sorry master (edited wrong comment)

[photodude]

Nice to see that memory failure is gone.

Am I seeing things right in that Jenkins reports a failure of the "JHtmlDateTest::testRelative" test? (ignoring the memcache(d) connection failures those should probably be skipped if there is no memcache(d) connection)

[rdeutz]

@photodude yes you are right, I looked into it and couldn't find out why, need some more time to investigate. If someone else has an idea feel free to give a hint.

[csthomas]

About memcache(-d) connection.

Why build can not run something lke:

memcached -l 127.0.0.1 -m64 -p 11211

for testing time from the same user as phpunit and after test kill it. Then code coverage report will contains memcache(-d) files.

[rdeutz]

really wondering what's going on, server is over 2.5 load average and I see a lot processes about email (imap, postmaster, dovecot, pop3???), not responding atm

[csthomas]

[UPDATED] I wrote a patch for that errors at #11900.

[csthomas]

If someone else has an idea feel free to give a hint.

Jenkins is too slow:)

We have to resign from @dataProvider dataTestRelative function and move it to testRelative or something similar.

I have added echo to that functions.

Result:

$ php libraries/vendor/phpunit/phpunit/phpunit
(DATA PROVIDER RUN AT 1472856540)PHPUnit 4.8.27 by Sebastian Bergmann and contributors.
Warning:        The Xdebug extension is not loaded
                No code coverage will be generated.

.....S.......................................................   61 / 5697 (  1%)
.............................................................  122 / 5697 (  2%)
...................................................(testRelative RUN AT 1472856542).(testRelative RUN AT 1472856542).(testRelative RUN AT 1472856542).(testRelative RUN AT 1472856542).(testRelative RUN AT 1472856542)......  183 / 5697 (  3%)
.............................................................  244 / 5697 (  4%)

[rdeutz]

You can't disable xdebug for the tests because then we don't get code coverage reports

[csthomas]

I only want to show you when DATA PROVIDER function is run and when testRelative is run. Time between them on Jenkins at build.joomla.org is more than 1 minute and there is a problem.

Above log is from my laptop and I do not need xdebug to test my PRs:)

[csthomas]

PR is ready #11900.

[csthomas]

What is decision about memcache(-d) on Jenkins? Blacklist or enable memcached server on default port?

[rdeutz]

working on it and try to figure out how to integrate it into the testing process, will report back

[csthomas]

For blacklist I sent an example of solution #11900.

[csthomas]

But the best way would be to enable as much extensions as you can:)

[rdeutz]

@csthomas have started it like you suggested and tests are running well, need to find out how I have to start it when I will stop it at the end of the test run

[csthomas]

IMHO you do not need to do anything except merge it. As your PR for JDateTest.

$_ENV['BUILD_TAG'] is almost the same as getenv('BUILD_TAG') and it is working now but only for a few class tests like: JCacheStorageMemcacheTest, JCacheStorageMemcacheTest.

JCacheTest and JCacheStorageTests does not use blacklists for now and generetes 11 errors.

[rdeutz]

closing this here because the issue is solved

[photodude]

Thanks everyone.