search

Shopify / ghostferry

The swiss army knife of live data migrations
https://shopify.github.io/ghostferry
MIT License
694 stars 65 forks source link

Skip target verification in copydb tutorial #254

Closed paarthmadan closed 2 years ago

paarthmadan commented 3 years ago

When running through the copydb tutorial, it errors out due to lack of correct permissions.

We don't SkipTargetVerification in the suggested config, therefore the target DB requires replication privileges, which aren't currently suggested (namely: REPLICATION CLIENT, REPLICATION SLAVE).

We should add SkipTargetVerification: true to the copydb tutorial to ensure that the permissions suggested are sufficient.

Further, the example configuration file uses the ghostferry user, which is the user we're instructed to create permissions for.

The actual example config listed in the source at examples/copydb/conf.json uses the root user. None of those grants are necessary then.

This PR fixes those issues:

  1. Use the ghostferry user as opposed to root.
  2. Skip target verification so that the tutorial doesn't error out on lack of permissions for the target.
paarthmadan commented 3 years ago

do we still need SUPER privileges as well?

It my local tests, I don't require SUPER privileges for the source or the target.

And also on the target?

The change in this PR ensures we list out REPLICATION SLAVE, REPLICATION CLIENT as some of the necessary privileges on the target, as our docs didn't say this prior.

shuhaowu commented 3 years ago

Wait. We need SUPER for the source if source is a read only replica, because we use FOR UPDATE.

shuhaowu commented 3 years ago

So we don't need those permissions if we set SkipTargetVerification: true in the example config...

In fact we might want to default it to that value for copydb, because copydb tends to run without target verification....

paarthmadan commented 3 years ago

@shuhaowu My mistake, I should've been clearer. For the copydb tutorial, the minimum permissions required are those mentioned. The source in the tutorial is not setup as a read-only replica, thus we don't need SUPER. Do you think this is okay?

I'll update the PR so that the copydb tutorial skips target verification (and thus does not require privileges for the target)

shuhaowu commented 3 years ago

The source in the tutorial is not setup as a read-only replica, thus we don't need SUPER. Do you think this is okay? I'll update the PR so that the copydb tutorial skips target verification (and thus does not require privileges for the target)

Both sounds good to me.

tiwilliam commented 3 years ago

These configs work fine with the committed docker compose + seed commands in docs, is this still the case after switching to the ghostferry user? If not I think we should create the user and grants on dev up.