backup: support multiple schedules (#5633)

This is an automated cherry-pick of #5633

What problem does this PR solve?

Close #5632

What is changed and how does it work?

In order to support multiple active schedulers, we need to co-ordinate the scheduling to make sure there is only one active backup within the same namespace. A new label is introduced in this PR, whose key is tidb.pingcap.com/backup-schedule-group. Backup schedule CRD can be assigned such a label, and the value is not empty, if it's in a multiple schedule group, otherwise, the schedule is independent. At backup schedule check time, all schedules in the same group need to be checked.

Here is a sample volume backup schedule CRD with the label specified tidb.pingcap.com/backup-schedule-group: ebs-backup-schedule-group1.

apiVersion: federation.pingcap.com/v1alpha1
kind: VolumeBackupSchedule
metadata:
  name: sche-rolling
  namespace: bc-fed-admin
  labels:
    tidb.pingcap.com/backup-schedule-group: ebs-backup-schedule-group1
spec:
  backupTemplate:
    clusters:
    - k8sClusterName: dataplane-a
      tcName: restore1-a
      tcNamespace: backup-a
    - k8sClusterName: dataplane-b
      tcName: restore1-b
      tcNamespace: backup-b
    - k8sClusterName: dataplane-c
      tcName: restore1-c
      tcNamespace: backup-c
    template:
      br:
        sendCredToTikv: false
      calcSizeLevel: disabled
      cleanPolicy: Delete
      resources: {}
      s3:
        bucket: wangle-ebs-test-us-west-2
        prefix: zm-rolling
        provider: aws
        region: us-west-2
      serviceAccount: tidb-backup-manager
      snapshotsDeleteRatio: 1
      toolImage: gcr.io/pingcap-public/zhongming/qa/br:240401173528
      volumeBackupInitJobMaxActiveSeconds: 1200
  maxReservedTime: 10m
  pause: true
  schedule: '*/10 * * * *'

This PR also provides the similar support to snapshot (non-EBS snapshot) backup schedule.

Code changes

[X] Has Go code change
[ ] Has CI related scripts change

Tests

[x] Unit test
[ ] E2E test
[ ] Manual test
[ ] No code

Side effects

[ ] Breaking backward compatibility
[ ] Other side effects:

Related changes

[ ] Need to cherry-pick to the release branch
[ ] Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

pingcap / tidb-operator