GuptaManan100
commented
1 year ago Trying to fix this issue isn't trivial. Here are my findings (and issues that have presented themselves).
So, we have a function call blocked on stream.SendMsg.
-
The first solution we explored was to be able to close the stream. Unfortunately, their so no way to do that from the server side. From the client side, just cancelling the context works but on the server, we don't have access to the cancel func to be able to do that. I looked at the grpc repository to see how to do that. The only thing that I found was at https://github.com/grpc/grpc-go/issues/465#issuecomment-259944377 where it is said that the only way to unblock is to call
stream.RecvMsg, but that too isn't possible on the server side since this Recv has to happen on the client stream. -
I then looked at the grpc docs. In one of the places https://grpc.io/docs/what-is-grpc/core-concepts/#cancelling-an-rpc, they say that a RPC can be cancelled from the client or the server, but they don't specify any way of doing that explicitly. https://grpc.io/docs/languages/go/basics/ has the docs on the different streaming rpcs and on how to call them. Here too, they don't mention how to stop a stream from the server. Maybe @vmg knows more about this. Me and @harshit-gangal couldn't figure this out.
-
Another possible fix is that we can just close the entire grpc server and restart it. According to the docs of grpc Server.Stop, https://pkg.go.dev/google.golang.org/grpc#Server.Stop this will close all active RPCs and then close the server. This could work for our use case. We can just restart the server in CheckMySQL.
-
Next possibiliy is to kill the vttablet process when CheckMySQL hangs. This can be done by an existing funciton
setTimeBomb. Currently the default is to not do this. We can parameterize the time-bomb value and just say that CheckMySQL being stuck will kill the vttablet after 2 minutes or whatever. (I personally don't think this is a good idea.) -
Final possibility is that we can actually create a separate go-routine which sends the message and in the callback wait for the go-routine to finish or a context that we create to be cancelled. If the context is cancelled, we can exit out and error. What this accomplishes is that the we continue to be blocked on
sendMsgbut now on a separate go-routine. So we can actually end the request and unblock CheckMySQL. Here are some code-snippets to show what I mean -// In StreamExecute ctx, cancel := context.WithCancel(ctx) defer cancel() tabletserver.ContextCancels = append(tabletserver.ContextCancels, cancel) err = q.server.StreamExecute(ctx, request.Target, request.Query.Sql, request.Query.BindVariables, request.TransactionId, request.ReservedId, request.Options, func(reply *sqltypes.Result) error { errChan := make(chan error) log.Errorf("Sending a stream now with %d rows", len(reply.Rows)) go func() { errChan <- stream.Send(&querypb.StreamExecuteResponse{ Result: sqltypes.ResultToProto3(reply), }) }() select { case err2 := <-errChan: log.Errorf("Finished sending a stream now, error = %v", err2) return err2 case <-ctx.Done(): return ctx.Err() } })
// And in closeAll we also call cancelContexts before waiting for empty requests var ContextCancels []context.CancelFunc
func cancelContexts() { for _, cancel := range ContextCancels { cancel() } }
The code above is just a poc, it isn't production grade. I don't even cleanup the cancel funcs after use. But I have verified that this indeed does unblock CheckMySQL.
Overview of the Issue
CheckMySQL can get indefinitely stuck if there is a long-running callback in StreamExecute.
The way that
StreamExecuterpc from vtgate works is as follows -vtgate and vttablet communicate via grpc and vttablets call
stream.Sendwhile vtgates call the counter-partstream.Recv.vtgates after receiving a result set, run the callback function on it. Let's say that the callback function is a long-running one and would take a lot of time.
vttablet on the other hand would keep getting results from MySQL and call
stream.Sendrepeatedly. According to the docs ofstream.Send-even though SendMsg doesn't block for the receiver to receive the message, it does block for flow control. So if the streaming query requires a lot of results to be sent, the first few calls don't block, but the next one does, because of flow control.
If MySQL crashes now, StreamExecute continues to stay blocked. Now since StreamExecute is blocked on this call, it doesn't finish its execution of
execRequest. This causes the wait for requests to be blocked until the callback function from vtgate doesn't complete. When it does, one more message is received, clearing more space in the buffer for StreamExecute. This causes StreamExecute to attempt to read one more packet from MySQL and finally fail unblocking everything.Reproduction Steps
select * from tableas the query and give it a very long callback. (Add time.Sleep to it, etc)Binary Version
Operating System and Environment details
Log Fragments
No response