E2E tests failing during deploys

xmtp / xmtp-node-go

Software for the nodes that currently form the XMTP network

MIT License

10 stars 3 forks source link

E2E tests failing during deploys #88

Closed snormore closed 1 year ago

snormore commented 2 years ago

Either an issue with the e2e runner not terminating cleanly or with node connectivity during deploys: https://app.datadoghq.com/dashboard/ask-2bf-idz/xmtpd-e2e

Screen Shot 2022-07-29 at 12 12 01 PM

Worth noting that container CPU alerts were triggered for all of the nodes during this time:

snormore commented 2 years ago

It looks like filter is dropping the connection sometimes, lining up with this cluster of e2e failures:

failed to dial 16Uiu2HAmSwQDoUmuALpn1aXQrH55g8ea2sffqARivKUobZzuAM7F:
  * [/ip4/127.0.0.1/tcp/36089/ws] dial tcp 127.0.0.1:36089: connect: connection refused
  * [/ip4/10.0.140.26/tcp/36089/ws] dial tcp 10.0.140.26:36089: connect: connection refused

snormore commented 2 years ago

Opened separate issue describing/tracking the root cause of this problem https://github.com/xmtp-labs/hq/issues/668

snormore commented 2 years ago

There are a few minor things left to do here after closing out https://github.com/xmtp-labs/hq/issues/668:

[x] In the waku test, if a deployment happens after the test has started and nodes selected, and the node becomes unavailable, then the test will fail, so it needs to be updated to be resilient to this (logs)
[ ] There's a panic in http client stream tear down that happens sometimes (logs) - seems to have been happening since mid Sept (logs)
[ ] In the messagev1 test, if a deployment happens while it's running and the node it's publishing to (via the LB) becomes unavailable, the request sometimes fails (logs)