search

Shopify / ghostferry

The swiss army knife of live data migrations
https://shopify.github.io/ghostferry
MIT License
694 stars 65 forks source link

schema fingerprint verifier #292

Open Manan007224 opened 3 years ago

Manan007224 commented 3 years ago

Introduction

Introducing Schema Verifier

This componnent currently collects source and target schema fingerprints periodically (1 minute by default) and verifies that the schema has not been changed on the source and target while the migration is still going on.

Technical details

I don't think there is anything special about the code written, most if it is self-explanotary.

Performace

Collecting Schema fingerprint every 1 minute should not affect performance in any sort for the migration as the interval is just too high, although I did some performance testing by setting the PeriodicallyVerifySchemaFingerprintInterval = 50ms, just for curiosity on how would the performance affect.

I collected for 30s cpuprofile (using pprof ) and offcpuprofile using (offcputime-bpfcc), but didn't notice any performance issues at all.

offcpu profile

offcpuprofile

cpu profile

profile001

shuhaowu commented 3 years ago

You imported some svg and png files. Is this intended?

Manan007224 commented 3 years ago

You imported some svg and png files. Is this intended?

Yeah, these files depict the performance graphs for offcpu and and cpu profiles in case anyone wants to take a look. Will delete it once the PR is ready to merge.

shuhaowu commented 3 years ago

You can just paste it directly in either a comment here or the PR description, would make it easier for people to see.

Manan007224 commented 3 years ago

You can just paste it directly in either a comment here or the PR description, would make it easier for people to see.

Have posted the CPU profile, but since the off CPU profile (basically flamegraph) is in svg format, I had to attach it as a file.

shuhaowu commented 3 years ago

in svg format, I had to attach it as a file.

What I usually do is either convert it into a rasterized format like png, or take a screenshot of the SVG and paste it in.