vitessio / vitess

Vitess is a database clustering system for horizontal scaling of MySQL.
http://vitess.io
Apache License 2.0
18.19k stars 2.06k forks source link

Bug Report: vtgate crashes when trying to run mysqldump #13780

Open gairadzi opened 10 months ago

gairadzi commented 10 months ago

Overview of the Issue

When running vtgate v16.0.* and trying run a mysqldump command against the vtgate it crashes with the following error:

runtime: goroutine stack exceeds 67108864-byte limit
runtime: sp=0xc004c7c380 stack=[0xc004c7c000, 0xc008c7c000]
fatal error: stack overflow
....

This happens on a fresh installation with an empty database

Reproduction Steps

  1. Deploy an example cluster:
    
    kubectl create ns vitess-test

kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_etcdlockservers.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessbackups.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessbackupstorages.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitesscells.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessclusters.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitesskeyspaces.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessshards.yaml

kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/priority.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/role.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/role_binding.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/service_account.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/operator.yaml

kubectl apply -n vitess-test -f https://gist.githubusercontent.com/gairadzi/f064e8cd31ddff3ff642b75d2bf9e2b3/raw/21afc35e6f617fd527bc699a783709d3cd953ac0/vitess-example-cluster.yaml


2. Download the certificates locally

kubectl get -n vitess-test secret example-cluster-config -o jsonpath='{ .data.cert.pem }' |base64 -D > cert.pem kubectl get -n vitess-test secret example-cluster-config -o jsonpath='{ .data.key.pem }' |base64 -D > key.pem


3. Create port forwarding to the vtgate service

kubectl port-forward -n vitess-test service/$(kubectl get -n vitess-test service -l planetscale.com/cell=app,planetscale.com/component=vtgate -o jsonpath='{ .items[0].metadata.name }') 3306


4. Run mysqldump command

mysqldump --skip-lock-tables --no-data --host 127.0.0.1 --user admin --password=admin --ssl-cert=cert.pem --ssl-key=key.pem main


### Binary Version

```sh
using the `vitess/lite:v16.0.3` image

Operating System and Environment details

using the `vitess/lite:v16.0.3` image

Log Fragments

https://gist.github.com/gairadzi/2e3e9b95ef47fa2947090cec24556b9d#file-vtgate-crash-log
dbussink commented 10 months ago

This is a v16 only issue, fix in https://github.com/vitessio/vitess/pull/13783.