search

PolymerLabs / arcs

Arcs
BSD 3-Clause "New" or "Revised" License
57 stars 35 forks source link

wdio tests may be exhibiting flakiness. #3402

Open shaper opened 5 years ago

shaper commented 5 years ago

My PR to alter empathy logging produced a Travis failure that looks unrelated. I haven't looked yet to see if this is a recurring or known issue.

@sjmiles as we recently updated wdio version so maybe it's related.

Starting ChromeDriver 75.0.3770.90 (a6dcaf7e3ec6f70a194cc25e8149475c6590e025-refs/branch-heads/3770@{#1003}) on port 4444
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[1564297310.383][SEVERE]: bind() failed: Cannot assign requested address (99)
[0-0] RUNNING in chrome - /shells/tests/specs/demo-test.js

Then we end up with:

[chrome  linux #0-0] 2 failing
[chrome  linux #0-0]
[chrome  linux #0-0] 1) demo pouchdb restaurants
[chrome  linux #0-0] timedout [20] waiting for "[title*="ou are free"]"
[chrome  linux #0-0] Error: timedout [20] waiting for "[title*="ou are free"]"
[chrome  linux #0-0]     at Timeout.setTimeout [as _onTimeout] (/home/travis/build/PolymerLabs/arcs/shells/tests/utils.js:107:14)
[chrome  linux #0-0]     at ontimeout (timers.js:436:11)
[chrome  linux #0-0]     at tryOnTimeout (timers.js:300:5)
[chrome  linux #0-0]     at listOnTimeout (timers.js:263:5)
[chrome  linux #0-0]     at Timer.processTimers (timers.js:223:10)
[chrome  linux #0-0]
[chrome  linux #0-0] 2) demo firebase restaurants
[chrome  linux #0-0] timedout [20] waiting for "[title*="ou are free"]"
[chrome  linux #0-0] Error: timedout [20] waiting for "[title*="ou are free"]"
[chrome  linux #0-0]     at Timeout.setTimeout [as _onTimeout] (/home/travis/build/PolymerLabs/arcs/shells/tests/utils.js:107:14)
[chrome  linux #0-0]     at ontimeout (timers.js:436:11)
[chrome  linux #0-0]     at tryOnTimeout (timers.js:300:5)
[chrome  linux #0-0]     at listOnTimeout (timers.js:263:5)
[chrome  linux #0-0]     at Timer.processTimers (timers.js:223:10)
shaper commented 5 years ago

Locally at ToT and synced to before my PR, I see these two errors:

[chrome  linux #0-0] 1) demo pouchdb gifts
[chrome  linux #0-0] timedout [20] waiting for "[title*="Check manufacturer"]"
[chrome  linux #0-0] Error: timedout [20] waiting for "[title*="Check manufacturer"]"
[chrome  linux #0-0]     at Timeout.setTimeout [as _onTimeout] (/usr/local/google/home/wkorman/workspace/arcs/shells/tests/utils.js:107:14)
[chrome  linux #0-0]     at ontimeout (timers.js:436:11)
[chrome  linux #0-0]     at tryOnTimeout (timers.js:300:5)
[chrome  linux #0-0]     at listOnTimeout (timers.js:263:5)
[chrome  linux #0-0]     at Timer.processTimers (timers.js:223:10)
[chrome  linux #0-0]
[chrome  linux #0-0] 2) demo firebase gifts
[chrome  linux #0-0] timedout [20] waiting for "[title*="Check manufacturer"]"
[chrome  linux #0-0] Error: timedout [20] waiting for "[title*="Check manufacturer"]"
[chrome  linux #0-0]     at Timeout.setTimeout [as _onTimeout] (/usr/local/google/home/wkorman/workspace/arcs/shells/tests/utils.js:107:14)
[chrome  linux #0-0]     at ontimeout (timers.js:436:11)
[chrome  linux #0-0]     at tryOnTimeout (timers.js:300:5)
[chrome  linux #0-0]     at listOnTimeout (timers.js:263:5)
[chrome  linux #0-0]     at Timer.processTimers (timers.js:223:10)
shaper commented 5 years ago

ToT is now green with a CRON build that ran ~10h ago:

https://travis-ci.org/PolymerLabs/arcs/builds/564839416

Are the new tests and/or test infra flaky? Going to leave this open for a bit to consider further.

sjmiles commented 5 years ago

Afaik,

[1564297310.383][SEVERE]: bind() failed: Cannot assign requested address (99)

happens just about every time on Travis, and has for a while. I've been conditioned to ignore it.

shaper commented 5 years ago

Notes that could fix that here.

Will retitle the issue to note wdio tests are exhibiting flakiness.

shaper commented 5 years ago

We are still seeing wdio flakiness on Travis. Two recent builds with examples:

@csilvestrini https://travis-ci.org/PolymerLabs/arcs/builds/583525796 @sherry-pra https://travis-ci.org/PolymerLabs/arcs/builds/583376167

The error now, however, looks different and solvable but I've not looked more closely, and I'm not sure why it would be flaky:

[0-0] 2019-09-10T22:01:02.207Z ERROR webdriver: Request failed due to session not created: session not created: This version of ChromeDriver only supports Chrome version 75
    at getErrorFromResponseBody (/home/travis/build/PolymerLabs/arcs/node_modules/webdriver/build/utils.js:371:10)
    at Request._callback (/home/travis/build/PolymerLabs/arcs/node_modules/webdriver/build/request.js:122:64)
    at Request.self.callback (/home/travis/build/PolymerLabs/arcs/node_modules/request/request.js:185:22)
    at Request.emit (events.js:189:13)
    at Request.EventEmitter.emit (domain.js:441:20)
    at Request.<anonymous> (/home/travis/build/PolymerLabs/arcs/node_modules/request/request.js:1161:10)
    at Request.emit (events.js:189:13)
    at Request.EventEmitter.emit (domain.js:441:20)
    at IncomingMessage.<anonymous> (/home/travis/build/PolymerLabs/arcs/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:277:13)
csilvestrini commented 5 years ago

I wish it would say which version of Chrome it's seeing, since it's not version 75...

My guess would be that either wdio was updated, or Chrome on Travis was updated. I'll see if I can find what version Travis has now.

csilvestrini commented 5 years ago

We use Chrome stable on Travis, which is currently version 77. The latest version of ChromeDriver is only version 76, but it seems to work with 77. I'll bump up the version number in package.json