status-im / team-core

Public repository for the Core team

1 stars 1 forks source link

RCA: Mailserver / protocol incompatibility 29/1/20 #9

Open rachelhamlin opened 4 years ago

rachelhamlin commented 4 years ago

Mailserver RCA - 29/1/2020

What

Protocol was incompatible
Clients could not talk to the server
- Nimbus PR was merged too soon, resulted in servers being disabled on client side
- Made changes in status-go on the mailserver side, not compatible with the client
Some envelopes had a bad timestamp that were making it to the server

How

We missed the failure in tests or these tests weren't run
Merge PR was flagged as Skip QA
We did run E2E tests and those passed
E2E test didn't have the environment variable that disables Nimbus set so it ran fine
Difficult to find the right combination of agents that contributed to the issue

Why

Wasn't clear that prod was going to be used for the RC

Moving forward

Document what version of the app uses what fleets (develop vs RC vs release)
Sanity check RC / Release builds on their respective fleets
Figure out why the environment variable didn't make it through on the tests and fix