search

backdrop / backdrop-issues

Issue tracker for Backdrop core.
144 stars 40 forks source link

[META] Fix random test failures #1478

Open jenlampton opened 8 years ago

jenlampton commented 8 years ago

There are several tests that fail randomly, usually only on the Travis CI infrastructure. Let's fix them, or replace them with others so that they don't do more harm than good.

Still giving us trouble:

Fixed:

quicksketch commented 8 years ago

I got a couple more randoms today. I think some of these new ones might be related to Background Fetch if the server isn't running fast enough. This was on PHP 5.4 and the tests ran for 30 minutes, compared to the normally speedy 10-15 minutes.


---- Bootstrap: Boot and exit hook invocation (HookBootExitTestCase) ----

Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      bootstrap.test     377 HookBootExitTestCase->testHookBootE
    Page cache cleared.
Fail      Other      bootstrap.test     380 HookBootExitTestCase->testHookBootE
    hook_boot called with aggressive cache and no cached page.
Fail      Other      bootstrap.test     381 HookBootExitTestCase->testHookBootE
    hook_exit called with aggressive cache and no cached page.

---- Layout: Layout Interface Test (LayoutInterfaceTest) ----

Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      layout.test        608 LayoutInterfaceTest->testMainConten
    "The "Main page content" block may only be added once to this
    layout." found
Exception Notice     layout.test        614 LayoutInterfaceTest->testMainConten
    Undefined offset: 0
Exception Notice     layout.test        618 LayoutInterfaceTest->testMainConten
    Undefined offset: 0
Fail      Other      layout.test        621 LayoutInterfaceTest->testMainConten
    Found the Save layout button
Fail      Other      layout.test        621 LayoutInterfaceTest->testMainConten
    Found the requested form fields at
Fail      Other      layout.test        622 LayoutInterfaceTest->testMainConten
    Layout saved successfully after removing old content block.
Fail      Other      layout.test        632 LayoutInterfaceTest->testMainConten
    Main content shown in sidebar after moving main content block.

---- Module: Module API (ModuleUnitTest) ----

Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      module.test         84 ModuleUnitTest->testModuleImplement
    The module implements cache is populated after requesting a page.
quicksketch commented 8 years ago

Details about the "User admin test case" fail:

---- User: User administration (UserAdminTestCase) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      user.test         1441 UserAdminTestCase->testUserAdmin() 
    User C blocked
Fail      E-mail     user.test         1450 UserAdminTestCase->testUserAdmin() 
    Activation mail sent to user C
quicksketch commented 8 years ago

Filed a PR at https://github.com/backdrop/backdrop/pull/1200 that addresses 4 of these failures. Updated the summary.

quicksketch commented 8 years ago

I think I've got all these worked out. For image styles and module API, the added sleep() timeouts should help the slower servers successfully pass when dealing with time-sensitive issues. The user and path alias tests I rewrote to use more robust XPath checking of elements instead of guessing based on the User ID.

The Layout: Layout Interface Test (LayoutInterfaceTest) is the only one on which I'm unclear. If that test fail keeps coming up, hopefully we will have at least narrowed the scope of the random test failures.

All the fixes so far have gone into 1.2.x and 1.x, so I'm tagging this for 1.2.4.

quicksketch commented 8 years ago

Here's dump of the current fail on the Layout Interface test fail report:

---- Layout: Layout Interface Test (LayoutInterfaceTest) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      layout.test        608 LayoutInterfaceTest->testMainConten
    "The "Main page content" block may only be added once to this
    layout." found
Exception Notice     layout.test        614 LayoutInterfaceTest->testMainConten
    Undefined offset: 0
Exception Notice     layout.test        618 LayoutInterfaceTest->testMainConten
    Undefined offset: 0
Fail      Other      layout.test        621 LayoutInterfaceTest->testMainConten
    Found the Save layout button
Fail      Other      layout.test        621 LayoutInterfaceTest->testMainConten
    Found the requested form fields at
Fail      Other      layout.test        622 LayoutInterfaceTest->testMainConten
    Layout saved successfully after removing old content block.
Fail      Other      layout.test        632 LayoutInterfaceTest->testMainConten
    Main content shown in sidebar after moving main content block.

Looks like our old friend "Invalid JSON" is still happening as well:

---- User: User email administration (UserAdminSettingsEmailFormTest) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Exception Uncaught e config.inc         898 ConfigFileStorage->read()          
    ConfigStorageReadException: The configuration file "user.mail" is
    not properly formatted JSON. in ConfigFileStorage->read() (line 898 of
    /home/travis/build/jenlampton/backdrop/core/includes/config.inc).
quicksketch commented 8 years ago

We're still getting a lot of failing page cache tests:

---- Bootstrap: Page cache test (BootstrapPageCacheTestCase) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------

Fail      Other      bootstrap.test     204 BootstrapPageCacheTestCase->testPag
    Cached page request.

I think this might have to do with the background fetch functionality and our use of sleep() to try and wait long enough, but that is not guaranteed, especially with the slower Travis CI servers. I filed a new PR at https://github.com/backdrop/backdrop/issues/1367.

quicksketch commented 8 years ago

I merged in https://github.com/backdrop/backdrop/pull/1367, which passed the fist attempt. We'll see how it does with new PRs as they come in.

Gormartsen commented 8 years ago

I am able to reproduce next issue with backdrop/backdrop#1367 applied.

Bootstrap: Page cache test (BootstrapPageCacheTestCase) 155 passes, 1 fail, 0 exceptions, and 41 debug messages

Test run duration: 58 sec

Detailed test results
---------------------

---- Bootstrap: Page cache test (BootstrapPageCacheTestCase) ----

Status    Group      Filename          Line Function
--------------------------------------------------------------------------------
Fail      Other      bootstrap.test     202 BootstrapPageCacheTestCase->testPag
    Initial page requests returned before shutdown functions are executed.

I am doing investigation here.

Gormartsen commented 8 years ago

PS: random fail happen again here: https://travis-ci.org/backdrop/backdrop/jobs/130238636

---- Bootstrap: Page cache test (BootstrapPageCacheTestCase) ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      bootstrap.test     328 BootstrapPageCacheTestCase->testPag
    Page was cached.
Gormartsen commented 8 years ago

This test is failing here:

  $start = microtime(TRUE);
    $this->backdropGet('system-test/sleep/shutdown/5', array(), $headers);
    $total = microtime(TRUE) - $start;
    $this->assertEqual($this->backdropGetHeader('X-Backdrop-Cache'), 'MISS', 'Initial page request was miss.');
    echo "Total: " . $total ."\n";
    $this->assertTrue($total < 5, 'Initial page requests returned before shutdown functions are executed.');

test with display $total has next result:

Bootstrap: Page cache test (BootstrapPageCacheTestCase) 155 passes, 1 fail, 0 exceptions, and 41 debug messages
Total: 5.0353920459747

Test run duration: 58 sec

Detailed test results
---------------------

---- Bootstrap: Page cache test (BootstrapPageCacheTestCase) ----

Status    Group      Filename          Line Function
--------------------------------------------------------------------------------
Fail      Other      bootstrap.test     203 BootstrapPageCacheTestCase->testPag
    Initial page requests returned before shutdown functions are executed.

It mean that 

 $this->backdropGet('system-test/sleep/shutdown/5', array(), $headers);

takes 5.03 sec to process

data from function system_test_sleep:

<div id="start">1463241968.3128</div>
  5 seconds will be slept at shutdown. 
<div id="end">1463241968.3128</div>
quicksketch commented 8 years ago

It mean that [...] takes 5.03 sec to process

Well in this case it's a "good" failure! Because the functionality is actually broken or not working. The page isn't being returned immediately, it's getting held up by a shutdown function when it shouldn't be. We've had some issues where backdrop_page_footer() doesn't flush and close the connection properly based on differing PHP configurations (e.g. mod_php vs. FastCGI vs. PHP-FPM). So this likely needs to be fixed in backdrop_page_footer() rather than adjusting the test.

Gormartsen commented 8 years ago

Thank you for details @quicksketch . It is different setup from travis. I am going to figure out this one.

Gormartsen commented 8 years ago

@quicksketch I fixed this (system-test/sleep/shutdown/5) issue. PR is here: https://github.com/backdrop/backdrop/pull/1378

quicksketch commented 8 years ago

It looks like my recent fixes to the page cache (adding sleep() and disabling the background fetch) missed the testPageCompression() test, which is failing regularly in both PHP 5.3 and 7.0.

---- Bootstrap: Page cache test (BootstrapPageCacheTestCase) ----

Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      bootstrap.test     328 BootstrapPageCacheTestCase->testPag
    Page was cached.

I've filed another PR at https://github.com/backdrop/backdrop/pull/1383. Perhaps combined with yours, we'll get the page cache tests fixed entirely.

quicksketch commented 8 years ago

Perhaps combined with yours, we'll get the page cache tests fixed entirely.

We haven't had any page cache failures since this change. Yay! Fewer failures. We are still getting several regularly though, let's keep at it.

quicksketch commented 8 years ago

PHP 7 gets this error fairly frequently: (fixed in https://github.com/backdrop/backdrop/issues/1423)

---- SimpleTest: SimpleTest functionality (SimpleTestFunctionalTest) ----

Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Exception Notice     simpletest.module  219 _simpletest_batch_finished()       
    Undefined offset: 0

And PHP 5.3 gets this one fairly regularly still: (fixed in https://github.com/backdrop/backdrop/pull/1450)

---- User: User email administration (UserAdminSettingsEmailFormTest) ----

Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Exception Uncaught e config.inc         898 ConfigFileStorage->read()          
    ConfigStorageReadException: The configuration file &quot;user.mail&quot; is
    not properly formatted JSON. in ConfigFileStorage-&gt;read() (line 898 of
    /home/travis/build/backdrop/backdrop/core/includes/config.inc).
klonos commented 8 years ago

I constantly seem to be getting this in the php70 tests of my PRs:

Layout: Layout BlockText Test (LayoutBlockTextTest)

---- Layout: Layout BlockText Test (LayoutBlockTextTest) ----
Status Group Filename Line Function
--------------------------------------------------------------------------------
Exception Notice     layout.test       1723 LayoutBlockTextTest->testBlockText(
    Undefined index: file
Fail      Other      layout.test       1723 LayoutBlockTextTest->testBlockText(
    File usage recorded for the file within the block.
Fail      Other      layout.test       1724 LayoutBlockTextTest->testBlockText(
    File has been marked permanent by its file usage.
Exception Notice     layout.test       1743 LayoutBlockTextTest->testBlockText(
    Undefined index: file
Fail      Other      layout.test       1743 LayoutBlockTextTest->testBlockText(
    File usage is unaffected by removing the block from the layout.
Fail      Other      layout.test       1744 LayoutBlockTextTest->testBlockText(
    File still exists and is marked as permanent after removing the block from
    the layout.
klonos commented 7 years ago

@quicksketch tiny coding standards fix, follow-up to your PR: https://github.com/backdrop/backdrop/pull/1945

quicksketch commented 7 years ago

Thanks @klonos! Merged that space fix to 1.x and 1.7.x :+1:

klonos commented 6 years ago

@quicksketch I am still getting very frequent fail comment-new class not found. comment.test:628 failures. Do you think perhaps worth increasing the time in https://github.com/backdrop/backdrop/blob/1.x/core/modules/comment/tests/comment.test#L575 ?

klonos commented 6 years ago

Increase from 2 to 3 seconds: https://github.com/backdrop/backdrop/pull/2357

quicksketch commented 5 years ago

I am still getting very frequent fail comment-new class not found. comment.test:628 failures.

I encountered this again today. It's worth a shot to increase the timeout I guess, though I don't know if it will have any impact. Merged https://github.com/backdrop/backdrop/pull/2357 into 1.x and 1.11.x.

jenlampton commented 5 years ago

Problems with BatchProcessingTestCase in PHP 5.3 only:

---- Batch API: Batch processing (BatchProcessingTestCase) ----

Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      batch.test         103 BatchProcessingTestCase->testBatchF
    Batches defined in separate submit handlers performed successfully.
jenlampton commented 5 years ago

Problems with CommentInterfaceTest in PHP 5.3 only:

---- Comment: Comment interface (CommentInterfaceTest) ----

Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      comment.test       628 CommentInterfaceTest->testCommentCl
    comment-new class not found.
quicksketch commented 5 years ago

New PR for fixing the comment-new race-condition that causes what might be our most common random failure: https://github.com/backdrop/backdrop/pull/2601

herbdool commented 5 years ago

RTBC for this PR https://github.com/backdrop/backdrop/pull/2601#issuecomment-485200847 @quicksketch

klonos commented 5 years ago

From time to time, I see this:

"Generated 20 URL aliases." found path_pattern.test:929

klonos commented 5 years ago

...never mind; was already in the issue summary as Path: Path pattern bulk updating (PathPatternBulkUpdateTestCase)

quicksketch commented 4 years ago

A new PR for a failure that's been happening frequently since 1.16.0: https://github.com/backdrop/backdrop/pull/3173

indigoxela commented 3 years ago

Seems like CommentInterfaceTest is back...

quicksketch commented 3 years ago

We've made a lot of progress recently since the switch to GitHub Actions (a lot of these had been showing up more frequently). The following now seem resolved:

The fixing of #5205 should fix intermittent failures in several other tests (anything that used batch processing). I think this may include #1965, but I'll leave that issue open to confirm.