Open gairadzi opened 10 months ago
When running vtgate v16.0.* and trying run a mysqldump command against the vtgate it crashes with the following error:
mysqldump
runtime: goroutine stack exceeds 67108864-byte limit runtime: sp=0xc004c7c380 stack=[0xc004c7c000, 0xc008c7c000] fatal error: stack overflow ....
This happens on a fresh installation with an empty database
kubectl create ns vitess-test
kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_etcdlockservers.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessbackups.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessbackupstorages.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitesscells.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessclusters.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitesskeyspaces.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessshards.yaml
kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/priority.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/role.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/role_binding.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/service_account.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/operator.yaml
kubectl apply -n vitess-test -f https://gist.githubusercontent.com/gairadzi/f064e8cd31ddff3ff642b75d2bf9e2b3/raw/21afc35e6f617fd527bc699a783709d3cd953ac0/vitess-example-cluster.yaml
2. Download the certificates locally
kubectl get -n vitess-test secret example-cluster-config -o jsonpath='{ .data.cert.pem }' |base64 -D > cert.pem kubectl get -n vitess-test secret example-cluster-config -o jsonpath='{ .data.key.pem }' |base64 -D > key.pem
3. Create port forwarding to the vtgate service
kubectl port-forward -n vitess-test service/$(kubectl get -n vitess-test service -l planetscale.com/cell=app,planetscale.com/component=vtgate -o jsonpath='{ .items[0].metadata.name }') 3306
4. Run mysqldump command
mysqldump --skip-lock-tables --no-data --host 127.0.0.1 --user admin --password=admin --ssl-cert=cert.pem --ssl-key=key.pem main
### Binary Version ```sh using the `vitess/lite:v16.0.3` image
using the `vitess/lite:v16.0.3` image
https://gist.github.com/gairadzi/2e3e9b95ef47fa2947090cec24556b9d#file-vtgate-crash-log
This is a v16 only issue, fix in https://github.com/vitessio/vitess/pull/13783.
Overview of the Issue
When running vtgate v16.0.* and trying run a
mysqldump
command against the vtgate it crashes with the following error:This happens on a fresh installation with an empty database
Reproduction Steps
kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_etcdlockservers.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessbackups.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessbackupstorages.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitesscells.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessclusters.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitesskeyspaces.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/crds/planetscale.com_vitessshards.yaml
kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/priority.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/role.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/role_binding.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/service_account.yaml kubectl apply -n vitess-test -f https://raw.githubusercontent.com/planetscale/vitess-operator/v2.9.3/deploy/operator.yaml
kubectl apply -n vitess-test -f https://gist.githubusercontent.com/gairadzi/f064e8cd31ddff3ff642b75d2bf9e2b3/raw/21afc35e6f617fd527bc699a783709d3cd953ac0/vitess-example-cluster.yaml
kubectl get -n vitess-test secret example-cluster-config -o jsonpath='{ .data.cert.pem }' |base64 -D > cert.pem kubectl get -n vitess-test secret example-cluster-config -o jsonpath='{ .data.key.pem }' |base64 -D > key.pem
kubectl port-forward -n vitess-test service/$(kubectl get -n vitess-test service -l planetscale.com/cell=app,planetscale.com/component=vtgate -o jsonpath='{ .items[0].metadata.name }') 3306
mysqldump --skip-lock-tables --no-data --host 127.0.0.1 --user admin --password=admin --ssl-cert=cert.pem --ssl-key=key.pem main
Operating System and Environment details
Log Fragments