Create a GitHub Actions job to run all Go MySQL tests with a master/replica configuration

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)

https://fleetdm.com

Other

3.11k stars 427 forks source link

Create a GitHub Actions job to run all Go MySQL tests with a master/replica configuration #19543

Open getvictor opened 5 months ago

getvictor commented 5 months ago

We have read-after-write DB consistency issues when running in environments with large replication lag, and we don't even know about them.

We can efficiently find many of these issues by running our entire test suite with a master DB + replica DB configuration. This run can be a separate job so we don't impact our CI time. If we use the same replication configuration for all tests, the tests can run in parallel (like they do for our current MySQL tests).

Bonus points:

Switch replication lag (master_delay) from 0 to 1 second (or 500ms, if possible)

Notes:

We can start with the integration tests, since they already supposed to simulate a real-world environment
We currently have a couple of master/replica tests in our CI. We would move them to the new run.

lukeheath commented 4 months ago

This is a good way to leverage existing tests by running our tests in an environment that more closely matches production deployments. Prioritizing for estimation.

sharon-fdm commented 4 months ago

Hey team! Please add your planning poker estimate with Zenhub @getvictor @jacobshandling @lucasmrod @mostlikelee @RachelElysia

noahtalerman commented 3 months ago

Hey @lukeheath I noticed that there's a couple engineering-initiated requests that made it through estimation but were not updated to use the user story template and didn't get a user story (this request, #19724, #19530). I think it's important that all stories get a user story so that any contributor can understand the problem.

If you agree, who is responsible for writing user stories for engineering-initiated requests before estimation? Is is the requestor? EM? I'm happy to do it as well.

lukeheath commented 3 months ago

@noahtalerman I agree, thanks for pointing this out. I'll make sure any eng-initiated stories are in user story format before prioritizing for estimation moving forward. It is the requestor's responsibility.

sharon-fdm commented 1 month ago

@getvictor, Is this 4.58.0 ? Should we remove the milestone if so?

getvictor commented 1 month ago

We committed this for this (4.57.0) sprint at kickoff.