[tidb-cluster chart] scheduled backup history is missing?

[danparisi]

I'd like to enable the scheduledBackup section in my tidb cluster chart and having the chance to restore some backup from a 7 days history, not necessarily the last one. It seems it is not possible if not setting up GCS or S3 strorage, but only a storageClassName.

Is there a way, or at least some workaround to have this feature in place?

[lonng]

@kennytm PTAL

[kennytm]

PTAL @weekface

[DanielZhangQD]

@danparisi For the history, you can update successfulJobsHistoryLimit: 3 in the tidb-cluster chart, it's 3 by default. In addition, even if the history pod is removed, the backup data is still there in the PVC <cluster_name>-scheduled-backup, you can check it's bound PV for all of the backup data.

[danparisi]

@DanielZhangQD okay so, the completed PODs I see in my namespace are actually the 3 dumps you mentioned, I understand now:

tidb-cluster-scheduled-backup-1587550200-vlscg   0/1     Completed   0          21h
tidb-cluster-scheduled-backup-1587550260-v2pbs   0/1     Completed   0          21h
tidb-cluster-scheduled-backup-1587600000-gmgc6   0/1     Completed   0          7h53m

But there's something I don't really get from the documentation. It says I can see all the backups by running: kubectl get pvc -n ${namespace} -l app.kubernetes.io/component=backup,pingcap.com/backup-cluster-name=${cluster_name} (please also note that there's an error in the command, as the component name is scheduled-backup)

But what I see is actually my PVC that is the same one for all the PODs in the history, isnt'it?

kubectl get pvc -l app.kubernetes.io/component=scheduled-backup,pingcap.com/backup-cluster-name=tidb-cluster
NAME                            STATUS   VOLUME
tidb-cluster-scheduled-backup   Bound    pvc-a5e52a4a-83f0-11ea-8e41-0050568cb224

So, in the end, what is the restore_name I should use for restoring a backup from the history?

helm install pingcap/tidb-backup --name=${restore_name} -f values.yaml --version=${version}

Maybe the POD name from the history (i.e. tidb-cluster-scheduled-backup-1587550200-vlscg)? And where can I find it it they are deleted from the POD list?

[DanielZhangQD]

But there's something I don't really get from the documentation. It says I can see all the backups by running: kubectl get pvc -n ${namespace} -l app.kubernetes.io/component=backup,pingcap.com/backup-cluster-name=${cluster_name} (please also note that there's an error in the command, as the component name is scheduled-backup)

This is for ad-hoc backups, not for scheduled backups, for ad-hoc backups, each backup will use a PVC. For scheduled backup, only one PVC for all backups. But what I see is actually my PVC that is the same one for all the PODs in the history, isnt'it?

kubectl get pvc -l app.kubernetes.io/component=scheduled-backup,pingcap.com/backup-cluster-name=tidb-cluster
NAME                            STATUS   VOLUME
tidb-cluster-scheduled-backup   Bound    pvc-a5e52a4a-83f0-11ea-8e41-0050568cb224

So, in the end, what is the restore_name I should use for restoring a backup from the history?

helm install pingcap/tidb-backup --name=${restore_name} -f values.yaml --version=${version}

Maybe the POD name from the history (i.e. tidb-cluster-scheduled-backup-1587550200-vlscg)? And where can I find it it they are deleted from the POD list?

To restore one of the scheduled backup, you can set below configs in the values.yaml of tidb-backup chart:

name: <clusterName>
scheduledBackupName: scheduled-backup-20190822-041004

You can check the PV that bound to the tidb-cluster-scheduled-backup PVC for all of the backups if you are using PVs in IDC. In addition, you can update successfulJobsHistoryLimit: 3 according to your backup frequency and make sure the Pods needed are preserved.

[danparisi]

Hi @DanielZhangQD Thanks for the answer, I almost managed to let it work.

This is the configuration I changed in my tidb-backup chart values.yaml:

• name: clusterName value from my tidb-cluster chart
• mode: scheduled-restore
• scheduledBackupName: name of the Completed POD backup (Keep in mind that backup are created by tidb-cluster scheduleBackup)

But it fails because of this error:

2020/04/27 12:46:21 loader.go:336: [error] [loader] scan dir[/data/tidb-cluster-scheduled-backup-1587990300-hxxd8] failed, err[invalid mydumper files for there are no `-schema-create.sql` files found]
2020/04/27 12:46:21 main.go:84: [fatal] /home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/loader/loader.go:433: invalid mydumper files for there are no `-schema-create.sql` files found
/home/jenkins/agent/workspace/build_tidb_enterprise_tools_master/go/src/github.com/pingcap/tidb-enterprise-tools/loader/loader.go:337: 

I saw this is qui a common issue many people had and I also found THIS HINT in your documentation. But in my scheduleBackup chart options -T or --tables-list are missing:

options: "-t 16 -r 10000 --skip-tz-utc --verbose=3"

And by the way I cannot manipulate data inside the backup volume

[DanielZhangQD]

@danparisi Sorry for the late response! scheduledBackupName is not the Pod name, but the directory name in the PV bound to the PVC of the scheduled backup. BTW, what's your environment of running TiDB, public cloud or private IDC? FYI, we do not maintain the tidb-cluster chart any more from TiDB Operator v1.1, we strongly recommend using custom resources to do backup and restore with S3/GCS with v1.1, you can check detail with mydumper or br.

[danparisi]

Hi @DanielZhangQD, thanks for the hint! I'll delete my tidb-cluster chart in favour of CRs.

Our k8s cluster is installed on a private cloud, so S3/GCS solutions are not fitting for me

[DanielZhangQD]

For S3, you may try ceph which is also supported. The backup and restore CRs do not support local PVCs currently. If with PVC, you should be able to check the backup directory in the PV that the scheduled backup using.

[danparisi]

As I said I'm not using S3 but only locale volumes.

If they are not supported in CRs I geuss I can use the tidb-backup chart for scheduled backup and restore

[DanielZhangQD]

OK, for scheduled backup, you have to use tidb-cluster chart.

[danparisi]

I'd not use that chart for scheduled backup only, It'd be an overkill. If there's no other solution I'll try to find something else.. I guess there are several solutions for MySQL that can be used here as well

[danparisi]

Anyway I gave it a try, and it seems I cannot use tidb-cluster for scheduledBackup feature only. It always pretends to run a tidb cluster and I cannot disabled it, but now I'm setting up the cluster via CRs..

[DanielZhangQD]

Anyway I gave it a try, and it seems I cannot use tidb-cluster for scheduledBackup feature only.

No, you cannot, it will always create a tidb cluster because the main feature of that chart is to create a tidb cluster. Actually, you can just get the two files for the scheduled backup under the charts/tidb-cluster/templates directory, update them and then create them manually:

ls charts/tidb-cluster/templates/scheduled-backup-*
charts/tidb-cluster/templates/scheduled-backup-cronjob.yaml charts/tidb-cluster/templates/scheduled-backup-pvc.yaml

[danparisi]

@DanielZhangQD thanks for the HINT, I already implemented a custom solution.

The ticket can be close, thanks for the support

[DanielZhangQD]

@danparisi Really sorry for the late response! Just feel free to feedback to us if any more issues.