search

nodejs / node

Node.js JavaScript runtime ✨🐢🚀✨
https://nodejs.org
Other
107k stars 29.27k forks source link

Investigate flaky tests on FreeBSD #8949

Closed mscdex closed 7 years ago

mscdex commented 8 years ago

I saw these three unrelated tests fail recently on CI: https://ci.nodejs.org/job/node-test-commit-freebsd/4650/nodes=freebsd10-64/console

not ok 129 parallel/test-child-process-fork-dgram
# TIMEOUT
  ---
  duration_ms: 60.55
not ok 546 parallel/test-http-server-consumed-timeout
# 
# assert.js:85
#   throw new assert.AssertionError({
#   ^
# AssertionError: Request timeout should not fire
#     at Object.exports.fail (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/common.js:443:10)
#     at IncomingMessage.req.setTimeout (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/parallel/test-http-server-consumed-timeout.js:13:12)
#     at emitOne (events.js:96:13)
#     at IncomingMessage.emit (events.js:188:7)
#     at Socket.<anonymous> (_http_server.js:310:50)
#     at emitNone (events.js:86:13)
#     at Socket.emit (events.js:185:7)
#     at Socket._onTimeout (net.js:342:8)
#     at ontimeout (timers.js:365:14)
#     at tryOnTimeout (timers.js:237:5)
  ---
  duration_ms: 1.943
not ok 970 parallel/test-timers-same-timeout-wrong-list-deleted
# 
# assert.js:85
#   throw new assert.AssertionError({
#   ^
# AssertionError: Elapsed time does not include second timer's timeout.
#     at process.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/parallel/test-timers-same-timeout-wrong-list-deleted.js:26:10)
#     at emitOne (events.js:101:20)
#     at process.emit (events.js:188:7)
  ---
  duration_ms: 1.69
jasnell commented 8 years ago

Seeing this also: https://ci.nodejs.org/job/node-test-commit-freebsd/4673/nodes=freebsd10-64/console

not ok 970 parallel/test-timers-same-timeout-wrong-list-deleted
# 
# assert.js:85
#   throw new assert.AssertionError({
#   ^
# AssertionError: Elapsed time does not include second timer's timeout.
#     at process.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/parallel/test-timers-same-timeout-wrong-list-deleted.js:26:10)
#     at emitOne (events.js:101:20)
#     at process.emit (events.js:188:7)
  ---
  duration_ms: 1.71
  ...
Trott commented 8 years ago

And this: https://ci.nodejs.org/job/node-test-commit-freebsd/4670/nodes=freebsd10-64/console

not ok 856 parallel/test-net-write-after-close
# TIMEOUT
# about to try to write
  ---
  duration_ms: 60.135
jbergstroem commented 8 years ago

Might be worth mentioning that I've seen increased amounts of processes lingering. Usually a bad sign as well as increasing the failure rate of future tests.

Trott commented 8 years ago

Yeah, I kind of suspect that when we see three tests failing, that the first one failed and didn't get cleaned up, and that's what's making the subsequent ones fail. Not always the case, but often enough...

jbergstroem commented 8 years ago

I cleaned a host and ran a test against master. Here's the failed tests:

not ok 131 parallel/test-child-process-fork-dgram
not ok 247 parallel/test-dgram-send-callback-multi-buffer
not ok 454 parallel/test-http-client-timeout-with-data
not ok 545 parallel/test-http-server-consumed-timeout
not ok 962 parallel/test-timers-blocking-callback

stalled processes after that run:

parallel/test-child-process-fork-dgram.js child