search

ipfs / ipfs-webui

A frontend for an IPFS Kubo and IPFS Desktop
https://webui.ipfs.io
MIT License
1.56k stars 488 forks source link

Flaky E2E tests #2065

Open lidel opened 1 year ago

lidel commented 1 year ago

This is a tracking issue for removing flakiness of E2E tests, as this impacts other repos.

I've been running tests locally in a loop until they fail (while npm test; do :; done) and identified 4 types of failures.

Pain points

(1)

[test:e2e  ]
[test:e2e  ]
[test:e2e  ] Running 31 tests using 4 workers
[test:e2e  ]
[test:e2e  ] Error: ENOENT: no such file or directory, open '/home/lidel/project/ipfs-webui/test/e2e/setup/ipfs-backend.json'
[test:e2e  ]
[test:e2e  ]    at setup/global-setup.js:7
[test:e2e  ]
[test:e2e  ]    5 | module.exports = async config => {
[test:e2e  ]    6 |   // Read and expose backend info in env availables inside of test() blocks
[test:e2e  ] >  7 |   const { rpcAddr, id, agentVersion } = JSON.parse(fs.readFileSync(path.join(__dirname, 'ipfs-backend.json')))
[test:e2e  ]      |                                                       ^
[test:e2e  ]    8 |   process.env.IPFS_RPC_ADDR = rpcAddr
[test:e2e  ]    9 |   process.env.IPFS_RPC_ID = id
[test:e2e  ]   10 |   process.env.IPFS_RPC_VERSION = agentVersion
[test:e2e  ]
[test:e2e  ]     at module.exports (/home/lidel/project/ipfs-webui/test/e2e/setup/global-setup.js:7:55)
[test:e2e  ]
[test:e2e  ]
[test:e2e  ]   31 skipped
[test:e2e  ]   1 error was not a part of any test, see above for details

(2)

[test:e2e  ]   ✘  5 ipns.test.js:28:5 › IPNS publishing › Settings screen › should list IPNS keys (30s)
[test:e2e  ]
[test:e2e  ]
[test:e2e  ]   1) ipns.test.js:28:5 › IPNS publishing › Settings screen › should list IPNS keys =================
[test:e2e  ]
[test:e2e  ]     Test timeout of 30000ms exceeded.
[test:e2e  ]
[test:e2e  ]     page.waitForSelector: Target closed
[test:e2e  ]     =========================== logs ===========================
[test:e2e  ]     waiting for selector "text=k51qzi5uqu5dlgkafkat85zoo7y2sfn66e9288hnvhiy7foqpics2zm8g19lm9" to be visible
[test:e2e  ]     ============================================================
[test:e2e  ]
[test:e2e  ]       135 |   for (const { id, name } of await ipfs.key.list()) {
[test:e2e  ]       136 |     if (name.startsWith('rm-key-test-')) continue // avoid race with removal tests
[test:e2e  ]     > 137 |     await page.waitForSelector(`text=${id}`)
[test:e2e  ]           |                ^
[test:e2e  ]       138 |     await page.waitForSelector(`text=${name}`)
[test:e2e  ]       139 |   }
[test:e2e  ]       140 | }
[test:e2e  ]
[test:e2e  ]         at waitForIPNSKeyList (/home/lidel/project/ipfs-webui/test/e2e/ipns.test.js:137:16)
[test:e2e  ]         at /home/lidel/project/ipfs-webui/test/e2e/ipns.test.js:30:7
[test:e2e  ]
[test:e2e  ]     Pending operations:
[test:e2e  ]       - page.waitForSelector at test/e2e/ipns.test.js:137:16
[test:e2e  ]
[test:e2e  ]
[test:e2e  ]
[test:e2e  ]   1 failed
[test:e2e  ]     ipns.test.js:28:5 › IPNS publishing › Settings screen › should list IPNS keys ==================
[test:e2e  ]   30 passed (38s)

(3)

4x more often than the rest

[test:unit ] FAIL src/bundles/peer-bandwidth.test.js
[test:unit ]   ● should sync added and removed peers
[test:unit ]
[test:unit ]     expect(received).toBe(expected) // Object.is equality
[test:unit ]
[test:unit ]     Expected: 149
[test:unit ]     Received: 97
[test:unit ]
[test:unit ]       161 |   bwPeers = store.selectPeerBandwidthPeers()
[test:unit ]       162 |
[test:unit ]     > 163 |   expect(bwPeers.length).toBe(nextPeers.length)
[test:unit ]           |                          ^
[test:unit ]       164 |
[test:unit ]       165 |   bwPeers.forEach(({ id }) => {
[test:unit ]       166 |     expect(nextPeers.some(p => p.peer === id)).toBe(true)
[test:unit ]
[test:unit ]       at Object.<anonymous> (src/bundles/peer-bandwidth.test.js:163:26)
[test:unit ]
[test:unit ] PASS src/bundles/node-bandwidth-chart.test.js
[test:unit ] PASS src/bundles/peer-locations.test.js
[test:unit ] PASS src/lib/dnd-backend.test.js
[test:unit ] PASS src/bundles/analytics.test.js
[test:unit ] PASS src/bundles/notify.test.js
[test:unit ] PASS src/bundles/files.test.js
[test:unit ] PASS src/lib/files.test.js
[test:unit ] PASS src/loader/AsyncRequestLoader.test.js
[test:unit ] PASS src/lib/count-dirs.test.js
[test:unit ]
[test:unit ] Test Suites: 1 failed, 1 skipped, 10 passed, 11 of 12 total
[test:unit ] Tests:       1 failed, 4 skipped, 59 passed, 64 total
[test:unit ] Snapshots:   0 total
[test:unit ] Time:        6.553 s
[test:unit ] Ran all test suites.
[test:e2e  ] [WebServer] (node:5070) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
[test:e2e  ] (Use `node --trace-deprecation ...` to show where the warning was created)

(4)

[test:unit ] Test Suites: 1 skipped, 11 passed, 11 of 12 total
[test:unit ] Tests:       4 skipped, 60 passed, 64 total
[test:unit ] Snapshots:   0 total
[test:unit ] Time:        7.077 s
[test:unit ] Ran all test suites.
[test:e2e  ] [WebServer] (node:6795) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
[test:e2e  ] (Use `node --trace-deprecation ...` to show where the warning was created)
  190.55s user 15.34s system 211% cpu 1:37.42 total
[test:unit ] PASS src/bundles/peer-bandwidth.test.js
[test:unit ] PASS src/bundles/identity.test.js
[test:unit ] PASS src/bundles/node-bandwidth-chart.test.js
[test:unit ] PASS src/bundles/peer-locations.test.js
[test:unit ] PASS src/bundles/analytics.test.js
[test:unit ] PASS src/loader/AsyncRequestLoader.test.js
[test:unit ] PASS src/bundles/notify.test.js
[test:unit ] PASS src/bundles/files.test.js
[test:unit ] PASS src/lib/files.test.js
[test:unit ] PASS src/lib/dnd-backend.test.js
[test:unit ] PASS src/lib/count-dirs.test.js
[test:unit ]
[test:unit ] Test Suites: 1 skipped, 11 passed, 11 of 12 total
[test:unit ] Tests:       4 skipped, 60 passed, 64 total
[test:unit ] Snapshots:   0 total
[test:unit ] Time:        7.163 s
[test:unit ] Ran all test suites.
[test:e2e  ] [WebServer] (node:10041) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
[test:e2e  ] (Use `node --trace-deprecation ...` to show where the warning was created)
ERROR: "test:e2e" exited with 1.
lidel commented 1 year ago

I'll try to bump playwright libs and re-run while npm test; do :; done see if it improves things or if the above problems still appear.

lidel commented 1 year ago

Seems that https://github.com/ipfs/ipfs-webui/pull/2066 stabilized test:e2e run via playwright, but we still have flaky test:unit tests:

[test:unit ] FAIL src/bundles/peer-bandwidth.test.js
[test:unit ]   ● should get bandwidth for added peers
[test:unit ]
[test:unit ]     expect(received).toBeTruthy()
[test:unit ]
[test:unit ]     Received: undefined
[test:unit ]
[test:unit ]       201 |   expect(bwPeers.length).toBe(peers.length)
[test:unit ]       202 |
[test:unit ]     > 203 |   bwPeers.forEach(({ bw }) => expect(bw).toBeTruthy())
[test:unit ]           |                                          ^
[test:unit ]       204 | })
[test:unit ]       205 |
[test:unit ]       206 | it('should periodically update bandwidth for peers', async () => {
[test:unit ]
[test:unit ]       at forEach (src/bundles/peer-bandwidth.test.js:203:42)
[test:unit ]           at Array.forEach (<anonymous>)
[test:unit ]       at Object.<anonymous> (src/bundles/peer-bandwidth.test.js:203:11)
[test:unit ]

The underlying problem with unit tests is that they don't do smart await until specific data is available, they just sleep for a specified amount:

https://github.com/ipfs/ipfs-webui/blob/f1389606a751e7536dc3956ca85346dd5b7d9a19/src/bundles/peer-bandwidth.test.js#L196-L197