search

vitessio / vitess

Vitess is a database clustering system for horizontal scaling of MySQL.
http://vitess.io
Apache License 2.0
18.42k stars 2.08k forks source link

Bug Report: RPCs to a new started VTTablet sometimes fail #14834

Closed GuptaManan100 closed 8 months ago

GuptaManan100 commented 9 months ago

Overview of the Issue

Starting a new vttablet and running RPC calls against it immediately start failing with the error - 

TabletManager.PrimaryStatus on aws_.... error: Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation (errno 1227) (sqlstate 42000) during query: SHOW MASTER STATUS: Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation (errno 1227) (sqlstate 42000) during query: SHOW MASTER STATUS

This is concerning since we added code to guard against this problem in https://github.com/vitessio/vitess/pull/14565. But it seems that there is a situation where we still end up in this problem.

Reproduction Steps

Run a vitess cluster and run RPCs against vttablet's right after thye come up.

Binary Version

main

Operating System and Environment details

-

Log Fragments

No response

GuptaManan100 commented 9 months ago

On investigation, me and @dbussink found that for the case when vttablets are running with restore_from_backup flag, the call to handleRestore causes WaitForDBAGrants to run asynchronously. In the meantime we start the RPC server and if an RPC arrives before that code has run, we can be in a situation where the users don't have the correct grants, but we have started running the RPCs.