search

nodejs / reliability

Make the CI green again
33 stars 9 forks source link

Potential flakes: 20180810 #11

Closed joyeecheung closed 5 years ago

joyeecheung commented 5 years ago

Produced with ncu-ci walk pr --stats, counting tests that failed more than 1 PR

Potential flaky tests:

-------------------------------- Stats --------------------------------
Reason     parallel/test-module-loading-globalpaths
Type       JS_TEST_FAILURE
Failed PR  5 (#21857, #22209, #22020, #22152, #22212)
Appeared   test-packetnet-ubuntu1604-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16260/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16304/
--------------------------------------------------------------------------------
Reason     parallel/test-tls-server-verify
Type       JS_TEST_FAILURE
Failed PR  5 (#22122, #21857, #22074, #22184, #21975)
Appeared   test-softlayer-alpine36_container-x64-1, test-digitalocean-alpine37_container-x64-2, test-softlayer-centos6-x64-2, test-softlayer-centos7-x64-1, test-digitalocean-debian8-x64-1, test-softlayer-debian8-x86-1, test-softlayer-debian9-x64-1, test-digitalocean-fedora26-x64-1, test-rackspace-fedora27-x64-1, test-digitalocean-ubuntu1404-x86-1, test-softlayer-ubuntu1404-x64-1, test-digitalocean-ubuntu1604-x86-2, test-rackspace-ubuntu1604-x64-1, test-joyent-ubuntu1710-x64-2, test-digitalocean-ubuntu1804_container-x64-2, test-digitalocean-freebsd11-x64-1, test-macstadium-macos10.11-x64-2, test-osuosl-ubuntu1404-ppc64_le-1, test-joyent-smartos16-x64-1, test-joyent-smartos17-x64-2, test-softlayer-ubuntu1604_sharedlibs_container-x64-5, test-digitalocean-ubuntu1604_sharedlibs_container-x64-6, test-joyent-ubuntu1604_sharedlibs_container-x64-3, test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-digitalocean-ubuntu1604_sharedlibs_container-x64-1, test-linuxonecc-rhel72-s390x-3, test-osuosl-aix61-ppc64_be-2, test-azure_msft-win10-x64-4, test-rackspace-win2008r2-x64-5, test-azure_msft-win2016-x64-5, test-rackspace-win2012r2-x64-4, test-requireio_mhdawson-debian9-armv6l_pi1p-1, test-requireio_jasnell-debian9-armv7l_pi2-1, test-requireio_pivotalagency-debian9-arm64_pi3-2, test-rackspace-debian8-x64-1, test-digitalocean-debian9-x64-1, test-rackspace-fedora26-x64-1, test-digitalocean-fedora27-x64-1, test-softlayer-ubuntu1404-x86-1, test-digitalocean-ubuntu1404-x64-1, test-joyent-ubuntu1710-x64-1, test-softlayer-ubuntu1804_container-x64-1, test-macstadium-macos10.11-x64-1, test-osuosl-ubuntu1404-ppc64_le-2, test-joyent-smartos16-x64-2, test-joyent-smartos17-x64-1, test-joyent-ubuntu1604_sharedlibs_container-x64-5, test-osuosl-aix61-ppc64_be-1, test-packetnet-ubuntu1604-x64-2, test-azure_msft-win10-x64-1, test-azure_msft-win2016-x64-3, test-requireio_bengl-debian9-armv6l_pi1p-1, test-requireio_notthetup_sayanee-debian9-arm64_pi3-1, test-softlayer-centos6-x64-1, test-rackspace-centos7-x64-1, test-digitalocean-ubuntu1604_sharedlibs_container-x64-7, test-softlayer-ubuntu1604_sharedlibs_container-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16259/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16266/
--------------------------------------------------------------------------------
Reason     parallel/test-trace-event-promises
Type       JS_TEST_FAILURE
Failed PR  4 (#22104, #22122, #22146, #22145)
Appeared   test-packetnet-centos7-arm64-2, test-scaleway-ubuntu1604-armv7l-2, test-packetnet-ubuntu1604-arm64-1, test-packetnet-centos7-arm64-1, test-scaleway-ubuntu1604-armv7l-1, test-packetnet-ubuntu1604-arm64-2, test-digitalocean-freebsd11-x64-1, test-digitalocean-alpine36_container-x64-1, test-digitalocean-alpine37_container-x64-2, test-softlayer-centos6-x64-2, test-softlayer-centos7-x64-1, test-rackspace-debian8-x64-1, test-softlayer-debian8-x86-1, test-digitalocean-debian9-x64-1, test-rackspace-fedora26-x64-1, test-rackspace-fedora27-x64-1, test-softlayer-ubuntu1404-x86-1, test-softlayer-ubuntu1404-x64-1, test-digitalocean-ubuntu1604-x86-1, test-rackspace-ubuntu1604-x64-1, test-joyent-ubuntu1710-x64-1, test-softlayer-ubuntu1804_container-x64-1, test-macstadium-macos10.11-x64-1, test-osuosl-ubuntu1404-ppc64_le-2, test-joyent-smartos16-x64-2, test-joyent-smartos17-x64-1, test-digitalocean-ubuntu1604_sharedlibs_container-x64-8, test-digitalocean-ubuntu1604_sharedlibs_container-x64-6, test-joyent-ubuntu1604_sharedlibs_container-x64-5, test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-digitalocean-ubuntu1604_sharedlibs_container-x64-1, test-scaleway-ubuntu1604-armv7l-3, test-linuxonecc-rhel72-s390x-3, test-osuosl-aix61-ppc64_be-1, test-azure_msft-win10-x64-1, test-rackspace-win2008r2-x64-6, test-azure_msft-win2016-x64-3, test-rackspace-win2012r2-x64-7, test-requireio_bengl-debian9-armv6l_pi1p-2, test-requireio_svincent-debian9-armv7l_pi2-2, test-requireio_williamkapke-debian9-arm64_pi3-3, test-osuosl-aix61-ppc64_be-2, test-joyent-smartos16-x64-1, test-joyent-smartos17-x64-2, test-softlayer-ubuntu1604_sharedlibs_container-x64-2, test-softlayer-ubuntu1604_sharedlibs_container-x64-5, test-joyent-ubuntu1604_sharedlibs_container-x64-3, test-digitalocean-debian8-x64-1, test-softlayer-debian9-x64-1, test-digitalocean-fedora26-x64-1, test-digitalocean-ubuntu1404-x86-1, test-digitalocean-ubuntu1604-x86-2, test-digitalocean-ubuntu1804_container-x64-2, test-osuosl-ubuntu1404-ppc64_le-1, test-macstadium-macos10.11-x64-2, test-azure_msft-win10-x64-5, test-rackspace-win2008r2-x64-3, test-azure_msft-win2016-x64-2, test-rackspace-win2012r2-x64-5
First CI   https://ci.nodejs.org/job/node-test-pull-request/16275/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16279/
--------------------------------------------------------------------------------
Reason     parallel/test-trace-events-fs-sync
Type       JS_TEST_FAILURE
Failed PR  4 (#22175, #22189, #22152, #22185)
Appeared   test-linuxonecc-rhel72-s390x-3, test-azure_msft-win10-x64-5
First CI   https://ci.nodejs.org/job/node-test-pull-request/16257/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16311/
--------------------------------------------------------------------------------
Reason     parallel/test-worker-dns-terminate
Type       JS_TEST_FAILURE
Failed PR  3 (#21875, #22189, #22198)
Appeared   test-osuosl-aix61-ppc64_be-2, test-digitalocean-ubuntu1604_sharedlibs_container-x64-10
First CI   https://ci.nodejs.org/job/node-test-pull-request/16268/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16285/
--------------------------------------------------------------------------------
Reason     sequential/test-benchmark-http
Type       JS_TEST_FAILURE
Failed PR  3 (#22145, #22209, #22106)
Appeared   test-digitalocean-fedora27-x64-1, test-packetnet-ubuntu1604-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16279/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16305/
--------------------------------------------------------------------------------
Reason     sequential/test-net-bytes-per-incoming-chunk-overhead
Type       JS_TEST_FAILURE
Failed PR  3 (#22104, #22074, #21875)
Appeared   test-digitalocean-ubuntu1404-x64-1
First CI   https://ci.nodejs.org/job/node-test-pull-request/16256/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16268/
--------------------------------------------------------------------------------
Reason     async-hooks/test-callback-error
Type       JS_TEST_FAILURE
Failed PR  2 (#22184, #22106)
Appeared   test-digitalocean-fedora27-x64-1
First CI   https://ci.nodejs.org/job/node-test-pull-request/16265/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16305/
--------------------------------------------------------------------------------
Reason     async-hooks/test-statwatcher
Type       JS_TEST_FAILURE
Failed PR  2 (#22122, #21975)
Appeared   test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-azure_msft-win2016-x64-4, test-joyent-ubuntu1710-x64-1
First CI   https://ci.nodejs.org/job/node-test-pull-request/16277/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16326/
--------------------------------------------------------------------------------
Reason     parallel/test-bootstrap-modules
Type       JS_TEST_FAILURE
Failed PR  2 (#22128, #21857)
Appeared   test-packetnet-ubuntu1604-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16236/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16260/
--------------------------------------------------------------------------------
Reason     parallel/test-fs-readfilesync-pipe-large
Type       JS_TEST_FAILURE
Failed PR  2 (#22020, #22152)
Appeared   test-packetnet-ubuntu1604-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16297/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16331/
--------------------------------------------------------------------------------
Reason     parallel/test-fs-syncwritestream
Type       JS_TEST_FAILURE
Failed PR  2 (#21857, #22020)
Appeared   test-packetnet-ubuntu1604-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16260/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16297/
--------------------------------------------------------------------------------
Reason     parallel/test-http-chunk-problem
Type       JS_TEST_FAILURE
Failed PR  2 (#22074, #22152)
Appeared   test-packetnet-ubuntu1604-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16261/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16303/
--------------------------------------------------------------------------------
Reason     parallel/test-http2-pipe
Type       JS_TEST_FAILURE
Failed PR  2 (#22128, #22020)
Appeared   test-rackspace-win2008r2-x64-4, test-rackspace-win2008r2-x64-3
First CI   https://ci.nodejs.org/job/node-test-pull-request/16253/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16297/
--------------------------------------------------------------------------------
Reason     parallel/test-stdio-pipe-redirect
Type       JS_TEST_FAILURE
Failed PR  2 (#22177, #22202)
Appeared   test-azure_msft-win10-x64-3, test-rackspace-win2008r2-x64-1, test-azure_msft-win2012r2-x64-2, test-azure_msft-win2012r2-x64-1, test-azure_msft-win2016-x64-2, test-rackspace-win2012r2-x64-9, test-azure_msft-win10-x64-5, test-azure_msft-win2016-x64-5
First CI   https://ci.nodejs.org/job/node-test-pull-request/16258/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16327/
--------------------------------------------------------------------------------
Reason     sequential/test-benchmark-net
Type       JS_TEST_FAILURE
Failed PR  2 (#22145, #22209)
Appeared   test-packetnet-ubuntu1604-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16279/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16296/
--------------------------------------------------------------------------------
Reason     sequential/test-benchmark-tls
Type       JS_TEST_FAILURE
Failed PR  2 (#21975, #22152)
Appeared   test-packetnet-ubuntu1604-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16254/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16303/

Build failures:

Reason     ERROR: Error fetching remote repo 'origin'
Type       JENKINS_FAILURE
Failed PR  2 (#22115, #22222)
Appeared   test-softlayer-centos6-x64-1, test-rackspace-centos7-x64-1, test-softlayer-ubuntu1404-x64-1, test-joyent-ubuntu1710-x64-2, test-softlayer-centos6-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16240/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16321/
--------------------------------------------------------------------------------
Reason     java.io.IOException: Backing channel JNLP4-connect connection from ... is disconnected.
Type       JENKINS_FAILURE
Failed PR  2 (#22104, #22222)
Appeared   test-packetnet-ubuntu1604-x64-2, test-rackspace-win2008r2-x64-5
First CI   https://ci.nodejs.org/job/node-test-pull-request/16275/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16321/

Linter failures (likely not a flaky, if not a build flake):

Reason     - /usr/home/iojs/build/workspace/node-test-linter/lib/dgram.js
Type       JS_TEST_FAILURE
Failed PR  4 (#22128, #21975, #21875, #22104)
Appeared   test-joyent-freebsd10-x64-2
First CI   https://ci.nodejs.org/job/node-test-pull-request/16253/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16256/
BridgeAR commented 5 years ago

I just updated the entries above to split them into three sections.

1) Flaky tests 2) Build related failures 3) Linter failures

BridgeAR commented 5 years ago

@nodejs/trace-events PTAL at: parallel/test-trace-event-promises and parallel/test-trace-events-fs-sync

@nodejs/http @nodejs/http2 PTAL at: parallel/test-http-chunk-problem, parallel/test-http2-pipe and sequential/test-benchmark-net

@nodejs/fs PTAL at: parallel/test-fs-readfilesync-pipe-large and parallel/test-fs-syncwritestream

@nodejs/workers PTAL at: parallel/test-worker-dns-terminate

@nodejs/modules PTAL at: parallel/test-module-loading-globalpaths

@nodejs/build-infra PTAL at the jenkins issues.

lundibundi commented 5 years ago

I think parallel/test-worker-dns-terminate needs a stress test. It should be fixed based on https://github.com/nodejs/node/issues/21327#issuecomment-411577291 and this PR https://github.com/nodejs/node/pull/22189 no longer fails after rebase and this one https://github.com/nodejs/node/pull/22198 fails but doesn't have a fix in it. Local --repeat 1920 on linux box have also passed just fine.

joyeecheung commented 5 years ago

@BridgeAR By the way I think if the failure is caused by the --worker job (most test-packetnet-ubuntu1604-x64-2 failures there), it probably needs help from @nodejs/workers instead - and most of them are actually ENOENT under --worker so it could have something to do with how tmpdir resolves there.

joyeecheung commented 5 years ago

Closing because..there is no reason to keep it open? By the way the --worker test failures reduced significantly these two days. I'll post the new results grabbed today.