search

backdrop-ops / backdropcms.org

Issue tracker for the BackdropCMS.org website
https://backdropcms.org
25 stars 21 forks source link

Upon restarting one test set (php53 or php70) through the zen.ci UI, a "-1" is appended to the URL of "details" link of the other test and you get "Page not found". #276

Open klonos opened 8 years ago

klonos commented 8 years ago

I had what seemed to be a random php70 failure in https://github.com/backdrop/backdrop/pull/1461

So I used the newly-added "restart" feature to restart the tests. This time php70 passed green, but php53 shows as still running after many hours. Clicking on the "details" link available next to the php53 tests, errors with "Page not found".

I realize that either restarting the tests (yet again) or closing and re-opening the PR might fix the issue, but I haven't done so yet because I thought that @Gormartsen might want to take a look at the situation in case it has any deeper bugs lurking.

Gormartsen commented 8 years ago

Thank you @klonos

This issue is related to https://github.com/Zen-CI/issues/issues/15

php53 tests passed properly for 1461. Correct link is here: https://zen.ci/backdrop/backdrop/test/test-php53-backdrop_backdrop_1461-2b969e9

I am working on solution.

klonos commented 8 years ago

Thanks @Gormartsen

For what it's worth, I closed and re-opened the PR and I think I am getting a consistent reproducible process so far... I got the same failure in php70:

Path: Path pattern bulk updating (PathPatternBulkUpdateTestCase) 52 passes, 1 fail, 0 exceptions, and 13 debug messages
...
Fail      Other      path_pattern.test  837 PathPatternBulkUpdateTestCase->test

...and I'm pretty sure that if I restart the tests, php70 will pass, but I'll get the php53 weirdness.

Gormartsen commented 8 years ago

Can you try to restart php7 only?

Sounds like this one is our random fail issue.

I assume that most of them connected with background processing feature. That's why it's not always repeatable and hard to reproduce.

klonos commented 8 years ago

Can you try to restart php7 only?

That's what I did the previous time. Did this again (clicked on the "details" link next to the php70 test and then on the restart button/tab), but it seems to restart the tests for both php70 and php53. ???

Gormartsen commented 8 years ago

Hm. Tnx. Will investigate it

klonos commented 8 years ago

Hello @Gormartsen The php53 test initially had the same path_pattern.test (random) failure and when I restarted it, it passed green but got the php70 test to get stuck "in full swing" status. As you can see in https://github.com/backdrop/backdrop/pull/1603 the php70 test has a wrong "details" link that 404s. So, this is the opposite situation than what was reported here initially.

klonos commented 8 years ago

I managed to reproduce this again in https://github.com/backdrop/backdrop/pull/1607 when you restart a failing test through "restart" button in the zen.ci UI, then the "details" link for the other, non-failing test gets a -1 at the end of its URL, making it 404. Removing that trailing -1 from the URL leads you to the actual test results/details page and all is green.

Now, because tests are restarted after suspicion of random failures, it is most likely that the test you restarted will come back green. But because the other test (that was previously passing) gets its status form the (wrong) URL in the details link, it is shown as permanently stuck in "in full swing" status.

klonos commented 8 years ago

@Gormartsen some more things that I've noticed that might help you troubleshoot this. First, note that I am getting this consistently with my PRs. Also note that all my PRs are done through the GitHub UI: I edit one file here on github.com and create a PR and then I update my PR by adding changes one file at a time to the branch on my personal forked repo.

The php70 test initially shows an error about failing to initiate. After a minute or so, the status changes to "Processing Tests". When I click on the details link, I notice that there is a series of "Prepare box", "Remove box", "Restart test" with reason "We lost connection to test box. Restart task." and finally "Prepare box" again. This time the sandbox gets a new IP.

Hope that all this info might be of some help.

klonos commented 8 years ago

So, initially I get box instance failures for both php53 and php70 tests:

github-pr_sandbox_failures_for_both_php53_and_php70

At this point, both "Details links" for the tests point to https://zen.ci/backdrop/backdrop/test/test-php53-backdrop_backdrop_1622-[hash]-1 and https://zen.ci/backdrop/backdrop/test/test-php70-backdrop_backdrop_1622-[hash]-1 and they 404.

...after a while, the php53 test recovers and its status turns to "Processing Tests" (now correctly pointing to https://zen.ci/backdrop/backdrop/test/test-php53-backdrop_backdrop_1622-[hash]), but the php70 test is still in "Test box instance failed" mode and the details link still points to the URL with the -1 at the end.

...after a while longer, both tests turn to "Processing Tests" and the details links point to the correct URL.

Waiting for the tests to finish, I then get random results: sometimes one of the tests will fail with random failures (usually the php70 test), sometimes both tests will be green.

klonos commented 6 years ago

...another example @Gormartsen: https://github.com/backdrop/backdrop/pull/2356

ghost commented 4 years ago

Is this still an issue needing to stay open @klonos? Just trying to clean up the queue a bit... :wink: