search

catapult-project / catapult

Deprecated Catapult GitHub. Please instead use http://crbug.com "Speed>Benchmarks" component for bugs and https://chromium.googlesource.com/catapult for downloading and editing source code..
https://chromium.googlesource.com/catapult
BSD 3-Clause "New" or "Revised" License
1.93k stars 562 forks source link

tracing.ui.tracks.drawing_container_perf_test.draw_softwareCanvas and tracing.ui.timeline_viewport_test.locationObj tests failing on canary #1745

Closed anniesullie closed 8 years ago

anniesullie commented 8 years ago

What I can reproduce locally (Mac/Linux):

tracing.ui.timeline_viewport_test.locationObj  failed (38.68ms) 
AssertionError: expected { viewX: 4, viewY: 1581 } to deeply equal { viewX: 4, viewY: 1566 }
    at Function.assert.deepEqual (http://127.0.0.1:8003/chai/chai.js:1:19923)
    at Object.testFn_ (http://127.0.0.1:8003/tracing/ui/timeline_viewport_test.html:134:12)
    at Object.TestCase.run (http://127.0.0.1:8003/tracing/base/unittest/test_case.html:79:19)
    at Object.<anonymous> (http://127.0.0.1:8003/tracing/base/unittest/test_runner.html:187:52)
    at Object.TestRunner.runCurrentTestCase_ (http://127.0.0.1:8003/tracing/base/unittest/test_runner.html:185:14)
    at Object.TestRunner.runOneTestCase_ (http://127.0.0.1:8003/tracing/base/unittest/test_runner.html:144:12)
    at runTask (http://127.0.0.1:8003/tracing/base/raf.html:64:21)
    at processRequests (http://127.0.0.1:8003/tracing/base/raf.html:93:9)

What I get on the buildbots: https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Linux%20Tryserver/builds/1209/steps/Tracing%20Dev%20Server%20Tests%3A%20Chrome%20Canary/logs/stdio

ERROR:root:EXCEPT  tracing.ui.timeline_viewport_test.locationObj AssertionError: expected { viewX: 4, viewY: 439 } to deeply equal { viewX: 4, viewY: 424 }
    at Function.assert.deepEqual (http://localhost:48454/chai/chai.js:1:19923)
    at Object.testFn_ (http://localhost:48454/tracing/ui/timeline_viewport_test.html:134:12)
    at Object.TestCase.run (http://localhost:48454/tracing/base/unittest/test_case.html:79:19)
    at Object.<anonymous> (http://localhost:48454/tracing/base/unittest/test_runner.html:187:52)
    at Object.TestRunner.runCurrentTestCase_ (http://localhost:48454/tracing/base/unittest/test_runner.html:185:14)
    at Object.TestRunner.runOneTestCase_ (http://localhost:48454/tracing/base/unittest/test_runner.html:144:12)
    at runTask (http://localhost:48454/tracing/base/raf.html:64:21)
    at processRequests (http://localhost:48454/tracing/base/raf.html:93:9)
FAILED  tracing.ui.timeline_viewport_test.locationObj 
EXCEPT  tracing.ui.tracks.drawing_container_perf_test.draw_softwareCanvas_Ten AssertionError: addHTMLOutput element has 0 height
    at f.HTMLTestCaseResult.addHTMLOutput (http://localhost:48454/tracing/base/unittest/html_test_results.html:176:7)
    at f.HTMLTestResults.addHTMLOutputForCurrentTest (http://localhost:48454/tracing/base/unittest/html_test_results.html:379:35)
    at Object.tr.b.unittest.addHTMLOutputForCurrentTest (http://localhost:48454/tracing/base/unittest/test_runner.html:41:22)
    at Object.TestCase.addHTMLOutput (http://localhost:48454/tracing/base/unittest/test_case.html:90:21)
    at Object.DCPerfTestCase.setUp (http://localhost:48454/tracing/ui/tracks/drawing_container_perf_test.html:51:12)
    at Object.GeneralDCPerfTestCase.setUp (http://localhost:48454/tracing/ui/tracks/drawing_container_perf_test.html:82:38)
    at Object.TestRunner.setUpCurrentTestCase_ (http://localhost:48454/tracing/base/unittest/test_runner.html:176:31)
    at Object.TestRunner.runOneTestCase_ (http://localhost:48454/tracing/base/unittest/test_runner.html:137:17)
    at runTask (http://localhost:48454/tracing/base/raf.html:64:21)
    at processRequests (http://localhost:48454/tracing/base/raf.html:93:9)
FAILED  tracing.ui.tracks.drawing_container_perf_test.draw_softwareCanvas_Ten 
EXCEPT  tracing.ui.tracks.drawing_container_perf_test.draw_softwareCanvas_AHundred AssertionError: addHTMLOutput element has 0 height
    at f.HTMLTestCaseResult.addHTMLOutput (http://localhost:48454/tracing/base/unittest/html_test_results.html:176:7)
    at f.HTMLTestResults.addHTMLOutputForCurrentTest (http://localhost:48454/tracing/base/unittest/html_test_results.html:379:35)
    at Object.tr.b.unittest.addHTMLOutputForCurrentTest (http://localhost:48454/tracing/base/unittest/test_runner.html:41:22)
    at Object.TestCase.addHTMLOutput (http://localhost:48454/tracing/base/unittest/test_case.html:90:21)
    at Object.DCPerfTestCase.setUp (http://localhost:48454/tracing/ui/tracks/drawing_container_perf_test.html:51:12)
    at Object.GeneralDCPerfTestCase.setUp (http://localhost:48454/tracing/ui/tracks/drawing_container_perf_test.html:82:38)
    at Object.TestRunner.setUpCurrentTestCase_ (http://localhost:48454/tracing/base/unittest/test_runner.html:176:31)
    at Object.TestRunner.runOneTestCase_ (http://localhost:48454/tracing/base/unittest/test_runner.html:137:17)
    at runTask (http://localhost:48454/tracing/base/raf.html:64:21)
    at processRequests (http://localhost:48454/tracing/base/raf.html:93:9)
FAILED  tracing.ui.tracks.drawing_container_perf_test.draw_softwareCanvas_AHundred

I'm going to revert my CL which enables testing on canary until this can be fixed.

@natduca @dj2

anniesullie commented 8 years ago

If anyone has a chance to look at this, please revert the revert of https://codereview.chromium.org/1427583010/ if you are able to fix.

natduca commented 8 years ago

I'm having trouble reproing on mac.. Annie can you help me with a repro today?

anniesullie commented 8 years ago

Sure! Here is what I got locally after syncing this morning:

$ tracing/bin/run_dev_server_tests --no-use-local-chrome --channel canary
Starting dev_server...
Starting Chrome 48.0.2560.0...
Waiting for tests to finish...
Killing Chrome...
ERROR:root:Tests failed!
ERROR:root:Server stderr:
ERROR:root:EXCEPT  tracing.ui.tracks.drawing_container_perf_test.draw_softwareCanvas_Ten AssertionError: addHTMLOutput element has 0 height
    at f.HTMLTestCaseResult.addHTMLOutput (http://localhost:62099/tracing/base/unittest/html_test_results.html:176:7)
    at f.HTMLTestResults.addHTMLOutputForCurrentTest (http://localhost:62099/tracing/base/unittest/html_test_results.html:379:35)
    at Object.tr.b.unittest.addHTMLOutputForCurrentTest (http://localhost:62099/tracing/base/unittest/test_runner.html:41:22)
    at Object.TestCase.addHTMLOutput (http://localhost:62099/tracing/base/unittest/test_case.html:90:21)
    at Object.DCPerfTestCase.setUp (http://localhost:62099/tracing/ui/tracks/drawing_container_perf_test.html:51:12)
    at Object.GeneralDCPerfTestCase.setUp (http://localhost:62099/tracing/ui/tracks/drawing_container_perf_test.html:82:38)
    at Object.TestRunner.setUpCurrentTestCase_ (http://localhost:62099/tracing/base/unittest/test_runner.html:176:31)
    at Object.TestRunner.runOneTestCase_ (http://localhost:62099/tracing/base/unittest/test_runner.html:137:17)
    at runTask (http://localhost:62099/tracing/base/raf.html:64:21)
    at processRequests (http://localhost:62099/tracing/base/raf.html:93:9)
FAILED  tracing.ui.tracks.drawing_container_perf_test.draw_softwareCanvas_Ten 
EXCEPT  tracing.ui.tracks.drawing_container_perf_test.draw_softwareCanvas_AHundred AssertionError: addHTMLOutput element has 0 height
    at f.HTMLTestCaseResult.addHTMLOutput (http://localhost:62099/tracing/base/unittest/html_test_results.html:176:7)
    at f.HTMLTestResults.addHTMLOutputForCurrentTest (http://localhost:62099/tracing/base/unittest/html_test_results.html:379:35)
    at Object.tr.b.unittest.addHTMLOutputForCurrentTest (http://localhost:62099/tracing/base/unittest/test_runner.html:41:22)
    at Object.TestCase.addHTMLOutput (http://localhost:62099/tracing/base/unittest/test_case.html:90:21)
    at Object.DCPerfTestCase.setUp (http://localhost:62099/tracing/ui/tracks/drawing_container_perf_test.html:51:12)
    at Object.GeneralDCPerfTestCase.setUp (http://localhost:62099/tracing/ui/tracks/drawing_container_perf_test.html:82:38)
    at Object.TestRunner.setUpCurrentTestCase_ (http://localhost:62099/tracing/base/unittest/test_runner.html:176:31)
    at Object.TestRunner.runOneTestCase_ (http://localhost:62099/tracing/base/unittest/test_runner.html:137:17)
    at runTask (http://localhost:62099/tracing/base/raf.html:64:21)
    at processRequests (http://localhost:62099/tracing/base/raf.html:93:9)
FAILED  tracing.ui.tracks.drawing_container_perf_test.draw_softwareCanvas_AHundred
anniesullie commented 8 years ago

Hmm, looks like the tracing.ui.timeline_viewport_test.locationObj failure does not repro anymore. Updating bug title. Sorry for the churn.

philippebeaudoin commented 8 years ago

I seem to be able to repro the timeline_viewport_test.locationObj consistently, let me take a look.

petrcermak commented 8 years ago

https://codereview.chromium.org/1433123002/: Out of curiosity, how much does the top of the track move? Is it a visible change? Is it by one pixel or more?

philippebeaudoin commented 8 years ago

It moves by 15 pixels (1566 to 1581). The rendering is still good, it's something to do with scroll bar appearing. I can repro consistently by hitting: http://127.0.0.1:8003/tracing/tests.html?testFilterString=locationObj&testSuiteName=tracing.ui.timeline_viewport_test

And resizing my browser window to ~10 pixels height. It doesn't happen with a normally sized browser window. Something to do with how we size the frame in which we run the test maybe? I'll take a look.

anniesullie commented 8 years ago

I have made no attempt to ensure the test runs Chrome with a reasonably sized window, apologies!

philippebeaudoin commented 8 years ago

That's just one way to trigger the bug. Another way (on my machine) is to have the test be far enough in the suite that it's offscreen when it runs. Further investigation shows that the problem is somewhat related to asyncTrack.scrollIntoView(); (Line 119 in timeline_viewport_test.html) If this line scrolls the window then getBoundingClientRect().top will change after asyncTrack.expanded = true;

I'm still trying to figure why.

philippebeaudoin commented 8 years ago

Here's maybe a more complete exposition of what I'm seeing when triggering the bug:

Does anybody know of a feature (or bug?) in the web platform that could cause this "looks like scrolling but scrollTop is unchanged" behavior?

philippebeaudoin commented 8 years ago

Naturally I just needed to post this to find that the 's scrollTop changes. This explains the scrolling behavior.

philippebeaudoin commented 8 years ago

Found a reasonable and relatively generic fix. https://codereview.chromium.org/1433123002/

@anniesullie Maybe we should pass the browser window height to the test bot, reduce one variable that may cause flake?

anniesullie commented 8 years ago

I think it is a good idea to have a fixed window size on the test bot. But I couldn't find anything on chrome command line switches. Do you know how to do it? @nedn does telemetry ever do this?

nedn commented 8 years ago

Telemetry uses "--window-size=1280,1024": https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/telemetry/internal/backends/chrome/desktop_browser_backend.py&l=209

anniesullie commented 8 years ago

Thanks, Ned! Sent https://codereview.chromium.org/1444673002/ to add that to the dev_server tests as well.

philippebeaudoin commented 8 years ago

Thanks Annie! With this and the proposed fix above I'm confident our screen-based tests will no longer be flaky.

anniesullie commented 8 years ago

Thanks, Philippe! But when I sync and run the following command, I still get 3 test failures, both on Mac and Linux. Do you see all tests passing?

tracing/bin/run_dev_server_tests --no-use-local-chrome --channel=canary

philippebeaudoin commented 8 years ago

Not sure if the previous patch fixes all your flakes. I never managed to repro some of these problems on my local machine. If everything looks fixed on your side can you close this?

I'll assign the bug to you to track re-enabling tests on the bots. If there's anything left, you can reassign to me.

anniesullie commented 8 years ago

All tests are passing for me at ToT in Linux and Mac, so closing this issue.

I sent out https://codereview.chromium.org/1459203002/ to enable the canary build for all dev_server tests.

natduca commented 8 years ago

Thank you so much @philippebeaudoin for driving this! This was a painful slog, but its going to be such a relief to not have canary- regressions any more catching us off guard!