backupjob for external MYSQL db is failing on init

[JannikZed]

I'm currently trying out stash to backup our external MYSQL database (hosted digitalocean one). Unfortunately, the job is silently failing. While analysing the issue I see a failing init container:

  initContainerStatuses:
  - containerID: docker://66870316ceadc56e4e5df0d2debb41cd4906cb5f128f4fdc35402c8f9dde9fe6
    image: stashed/stash-mysql:8.0.14
    imageID: docker-pullable://stashed/stash-mysql@sha256:276923201a8bc48fea40c7c43983ce5607983b6318c902d701e95779f9ccadd3
    lastState: {}
    name: mysql-backup-8-0-14-0
    ready: false
    restartCount: 0
    state:
      terminated:
        containerID: docker://66870316ceadc56e4e5df0d2debb41cd4906cb5f128f4fdc35402c8f9dde9fe6
        exitCode: 2
        finishedAt: "2020-02-24T11:50:10Z"
        reason: Error
        startedAt: "2020-02-24T11:50:10Z"

the container that gets created by the operator looks like this:

 initContainers:
  - args:
    - backup-mysql
    - --provider=s3
    - --bucket=XXXXXXX
    - --endpoint=nyc3.digitaloceanspaces.com
    - --region=
    - --path=stash-backup/mysql
    - --secret-dir=/etc/repository/secret
    - --scratch-dir=/tmp
    - --enable-cache=true
    - --max-connections=0
    - --hostname=host-0
    - --mysql-args=--all-databases
    - --namespace=backup
    - --appbinding=mysql-backup
    - --retention-keep-last=20
    - --retention-keep-hourly=0
    - --retention-keep-daily=0
    - --retention-keep-weekly=0
    - --retention-keep-monthly=0
    - --retention-keep-yearly=0
    - --retention-keep-tags=
    - --retention-prune=true
    - --retention-dry-run=false
    - --output-dir=/tmp/output

For sure, I also have a repository and an appbinding:

apiVersion: stash.appscode.com/v1alpha1
kind: Repository
metadata:
  name: do-s3-repo
  namespace: backup
spec:
  backend:
    s3:
      endpoint: nyc3.digitaloceanspaces.com # use server URL for s3 compatible other storage service
      bucket: stash-XXXXXX
      prefix: stash-backup/mysql
    storageSecretName: do-s3-backup-auth
---
apiVersion: appcatalog.appscode.com/v1alpha1
kind: AppBinding
metadata:
  name: mysql-backup
  namespace: backup
spec:
  clientConfig:
    url: mysql://private-db-mysXXXXXXXXXXXXX.ondigitalocean.com:25060/
  secret:
    name: do-managed-mysql-admin
  type: mysql

What am I doing wrong? Furthermore, why isn't there any issue visible? Kubernetes shows the backup session as running all the time:

NAME                             INVOKER-TYPE          INVOKER-NAME          PHASE     AGE
mysql-backup-config-1582545005   BackupConfiguration   mysql-backup-config   Running   9m30s
mysql-backup-config-1582545308   BackupConfiguration   mysql-backup-config   Running   4m27s

Any assistance highly appreciated ;)

[hossainemruz]

@JannikZed What is your Stash version?

[hossainemruz]

What is the output of the following command?

kubectl logs -n backup <backup job's pod> --all-containers

[JannikZed]

Hi there, I just installed stash with version 0.9.0 rc4. The logs:

I0224 12:05:08.541173       1 log.go:172] FLAG: --alsologtostderr="false"
I0224 12:05:08.541940       1 log.go:172] FLAG: --appbinding="mysql-backup"
I0224 12:05:08.542160       1 log.go:172] FLAG: --bucket="stash-prod-backup"
I0224 12:05:08.542361       1 log.go:172] FLAG: --enable-analytics="true"
I0224 12:05:08.542566       1 log.go:172] FLAG: --enable-cache="true"
I0224 12:05:08.542791       1 log.go:172] FLAG: --endpoint="nyc3.digitaloceanspaces.com"
I0224 12:05:08.543040       1 log.go:172] FLAG: --help="false"
I0224 12:05:08.543338       1 log.go:172] FLAG: --hostname="host-0"
I0224 12:05:08.543566       1 log.go:172] FLAG: --kubeconfig=""
I0224 12:05:08.543763       1 log.go:172] FLAG: --log-flush-frequency="5s"
I0224 12:05:08.543967       1 log.go:172] FLAG: --log_backtrace_at=":0"
I0224 12:05:08.544166       1 log.go:172] FLAG: --log_dir=""
I0224 12:05:08.544368       1 log.go:172] FLAG: --logtostderr="true"
I0224 12:05:08.544566       1 log.go:172] FLAG: --master=""
I0224 12:05:08.544753       1 log.go:172] FLAG: --max-connections="0"
I0224 12:05:08.544945       1 log.go:172] FLAG: --mysql-args="--all-databases"
I0224 12:05:08.545156       1 log.go:172] FLAG: --namespace="backup"
I0224 12:05:08.545367       1 log.go:172] FLAG: --output-dir="/tmp/output"
I0224 12:05:08.545585       1 log.go:172] FLAG: --path="stash-backup/mysql"
I0224 12:05:08.545772       1 log.go:172] FLAG: --provider="s3"
I0224 12:05:08.545975       1 log.go:172] FLAG: --region=""
I0224 12:05:08.546194       1 log.go:172] FLAG: --retention-dry-run="false"
I0224 12:05:08.546380       1 log.go:172] FLAG: --retention-keep-daily="0"
I0224 12:05:08.546615       1 log.go:172] FLAG: --retention-keep-hourly="0"
I0224 12:05:08.546806       1 log.go:172] FLAG: --retention-keep-last="20"
I0224 12:05:08.547139       1 log.go:172] FLAG: --retention-keep-monthly="0"
I0224 12:05:08.547339       1 log.go:172] FLAG: --retention-keep-tags="[]"
I0224 12:05:08.547542       1 log.go:172] FLAG: --retention-keep-weekly="0"
I0224 12:05:08.547743       1 log.go:172] FLAG: --retention-keep-yearly="0"
I0224 12:05:08.547967       1 log.go:172] FLAG: --retention-prune="true"
I0224 12:05:08.548167       1 log.go:172] FLAG: --scratch-dir="/tmp"
I0224 12:05:08.548363       1 log.go:172] FLAG: --secret-dir="***REDACTED***"
I0224 12:05:08.548576       1 log.go:172] FLAG: --service-name="stash-operator"
I0224 12:05:08.548761       1 log.go:172] FLAG: --stderrthreshold="0"
I0224 12:05:08.548973       1 log.go:172] FLAG: --v="0"
I0224 12:05:08.549211       1 log.go:172] FLAG: --vmodule=""
W0224 12:05:08.762857       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x130995e]

goroutine 1 [running]:
stash.appscode.dev/mysql/pkg.(*mysqlOptions).backupMySQL(0xc00044e240, 0xc0007355e0, 0x0, 0x0)
    /src/pkg/backup.go:166 +0x44e
stash.appscode.dev/mysql/pkg.NewCmdBackup.func1(0xc0002cca00, 0xc0002d4000, 0x0, 0x17, 0x0, 0x0)
    /src/pkg/backup.go:74 +0x185
github.com/spf13/cobra.(*Command).execute(0xc0002cca00, 0xc0001cfe00, 0x17, 0x18, 0xc0002cca00, 0xc0001cfe00)
    /src/vendor/github.com/spf13/cobra/command.go:826 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0xc0002cc500, 0x2, 0x0, 0x0)
    /src/vendor/github.com/spf13/cobra/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
    /src/vendor/github.com/spf13/cobra/command.go:864
main.main()
    /src/cmd/stash-mysql/main.go:40 +0x7b
Error from server (BadRequest): container "update-status-1" in pod "stash-backup-mysql-backup-config-1582545904-0-xf8pb" is waiting to start: PodInitializing

[hossainemruz]

Please uninstall previous Stash version and try this installing by this:

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update

$ helm install stash-operator appscode/stash \
  --version v0.9.0-rc.4 \
  --namespace kube-system \
  --set operator.tag=v0.9.0-rc.6

[JannikZed]

I tried it out and the job container is now running with image appscode/stash:v0.9.0-rc.6 - still I get the same error.

[hossainemruz]

Can you please show the YAML of your AppBinding?

[JannikZed]

sorry, I copied it in one file on issue creation. Here again:

apiVersion: appcatalog.appscode.com/v1alpha1
kind: AppBinding
metadata:
  name: mysql-backup
  namespace: backup
spec:
  clientConfig:
    url: mysql://private-db-mysXXXXXXXXXXXXX.ondigitalocean.com:25060/
  secret:
    name: do-managed-mysql-admin
  type: mysql

[hossainemruz]

Yeah. That's the problem. We haven't handled providing url instead of service reference in client-config yet.

[hossainemruz]

In this case, you have to create a Service pointing to your external DB and use that Service in AppBinding.

These articles might help:

• Kubernetes best practices: mapping external services
• Defining a Service for an External Database

[JannikZed]

ok! I just got it out of the docs.. let me try it with a service now. I created a service called do-managed-mysql that is just pointing to the URL private-db-mysXXXXXXXXXXXXX.ondigitalocean.com.

I get now the following output:

I0224 14:15:18.722958       1 commands.go:365] sh-output: null

I0224 14:15:18.723717       1 commands.go:120] Backing up stdin data
[golang-sh]$ mysqldump -u backup-user -h do-managed-mysql --all-databases | /bin/restic_0.9.6 backup --stdin --quiet --json --stdin-filename dumpfile.sql --host host-0 --cache-dir /tmp/restic-cache --cleanup-cache
mysqldump: Got error: 2003: Can't connect to MySQL server on 'do-managed-mysql' (110) when trying to connect
{"message_type":"error","error":{"Op":"read","Path":"/dumpfile.sql","Err":{}},"during":"archival","item":"/dumpfile.sql"}
Fatal: unable to save snapshot: snapshot is empty

seems like the backup job is not trying to use the correct hostname?

[JannikZed]

Appbinding like this:

apiVersion: appcatalog.appscode.com/v1alpha1
kind: AppBinding
metadata:
  name: mysql-backup
  namespace: backup
spec:
  clientConfig:
    service:
      name: do-managed-mysql
      port: 25060
      scheme: mysql
    # url: mysql://private-db-mysql-fra1-28488-prod-do-user-7029955-0.db.ondigitalocean.com:25060/
  secret:
    name: do-managed-mysql-admin
  type: mysql

[hossainemruz]

So, now Stash is unable to connect with the MySQL. Can you please make sure that the database is accessible from inside the cluster using that Service?

[JannikZed]

I was trying around a little bit and to me it just seems like you are forgetting to use the port settings. I started a mysql-client pod in the same namespace and copied exactly this command: mysqldump -u backup-user -h do-managed-mysql --all-databases- which is not working. The moment I'm adding a -P 25060 and a -p it's working just fine.