search

apache / solr-operator

Official Kubernetes operator for Apache Solr
https://solr.apache.org/operator
Apache License 2.0
246 stars 111 forks source link

Could not find a backup repository with name `solr-backup-s3-repo` #593

Closed anilkhichar closed 1 year ago

anilkhichar commented 1 year ago

Please suggest any pointers for debugging.

Is there any API where we can get the list of SolrOperator registered backup repositories? Also, for S3 backup, does the Operator first download the backup data on disk and then execute the S3 cp/sync?

Any internal detail would be helpful to debug the core issue.

Error:

2023-07-18T23:18:54.905Z        
ERROR   controller-runtime.manager.controller.solrbackup        
Error while taking SolrCloud backup     
{"reconciler group": "solr.apache.org", 
"reconciler kind": "SolrBackup",
 "name": "solr-backup", 
"namespace": "ns-solrcloud-int", 
"error": "Received bad response code of 500 from solr with response: 
{\n  \"responseHeader\":
{\n    \"status\":500,\n    \"QTime\":0},\n  \"error\":{\n    
\"msg\":\"Could not find a backup repository with name solr-backup-s3-repo\",
\n    \"trace\":\"java.lang.NullPointerException: 
Could not find a backup repository with name solr-backup-s3-repo\\
n\\tat 
java.base/java.util.Objects.requireNonNull(Unknown Source)\\n\\
tat org.apache.solr.core.backup.repository.
BackupRepositoryFactory.newInstance(BackupRepositoryFactory.java:74)
HoustonPutman commented 1 year ago

What version of Solr are you running? The Solr Operator just enables use of the existing Solr Backup repositories: https://solr.apache.org/guide/solr/latest/deployment-guide/backup-restore.html#backuprestore-storage-repositories

The S3 Backup Repository was added in 8.10 or 8.9, and is not supported for versions before that.

anilkhichar commented 1 year ago

Thank you @HoustonPutman for looking into this.

We are using Solr 9.2.1 with SolrOperator 0.5.1.

HoustonPutman commented 1 year ago

Can you provide your full SolrCloud specification? Are you using a custom solr xml, if so then you will need to add the backup repository yourself.

anilkhichar commented 1 year ago

Thank you @HoustonPutman for the guidance. Yes, we are using custom solr xml and that was the root cause. I wish, the solr backup document would have mentioned it. I will raise PR for that separately.

The Resolution:

  1. Configure S3 jars/lib path in the sharedPath
  2. Configure S3 repository in the custom solr xml 
    <?xml version="1.0" encoding="UTF-8" ?>
<solr>
  <str name="sharedLib">${solr.sharedLib:},${solr.install.dir}/contrib/s3-repository/lib</str>
  <solrcloud>
  .....
  </solrcloud>
  <backup>
    <repository name="solr-backup-s3-repo" class="org.apache.solr.s3.S3BackupRepository">
      <str name="s3.bucket.name">solr-backup-s3-bucket</str>
      <str name="s3.region">us-east-1</str>
    </repository>
  </backup>
</solr>

  3. Then on top it, it’s regular Solr Operator backup/restore configurations:

    kubectl edit solrcloud my-solr-cloud

solrBackup:
  enabled: "true"
  backupRepositories:
    name: "solr-backup-s3-repo"
    region: "us-east-1"
    bucket: "solr-backup-s3-bucket"
  4. Then on top it, it’s regular Solr Operator backup/restore configurations: 
    apiVersion: solr.apache.org/v1beta1
kind: SolrBackup
metadata:
  name: dev-solr-backup
  namespace: solr-workload-namespace
spec: 
  repositoryName: "solr-backup-s3-repo"
  solrCloud: "my-solr-cloud"
  collections: # upgraded solr operator versions can backup all available collections
    - my-collection
  recurrence: # Store one backup daily, and keep a week at a time.
    schedule: "@daily"
    maxSaved: 7
    disabled: false
HoustonPutman commented 1 year ago

Thanks for such good documentation!

Yes, we should certainly have a section listing all of the features that do not work by default when using a custom solr.xml, and how to get them working. That can go in the docs under the custom solr.xml section.

sbbagal13 commented 1 year ago

Hi @anilkhichar , I am also facing similar issue while configuring Solr backup , Below is error details, I am not using any custom solr.xml file. I have solr 8.11 and solr operator 0.5.0 , Could you please suggest any solution for it

2023-07-28T20:23:23.504Z ERROR controller-runtime.manager.controller.solrbackup Error while taking SolrCloud backup {"reconciler group": "solr.apache.org", "reconciler kind": "SolrBackup", "name": "solr-backup", "namespace": "cornerstone", "error": "**Unable to find backup repository to use for backup [solr-backup] (which specified the repository [s3-backups-1]). solrcloud must define a repository matching that name (or have only 1 repository defined)**."} github.com/go-logr/zapr.(*zapLogger).Error /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132 github.com/apache/solr-operator/controllers.(*SolrBackupReconciler).Reconcile /workspace/controllers/solrbackup_controller.go:133 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216 k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1 /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 k8s.io/apimachinery/pkg/util/wait.BackoffUntil /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 k8s.io/apimachinery/pkg/util/wait.JitterUntil /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185 k8s.io/apimachinery/pkg/util/wait.UntilWithContext /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99

HoustonPutman commented 1 year ago

You need to post your SolrCloud and SolrBackup yamls, or else there is no way we can help.

sbbagal13 commented 1 year ago

Thanks @HoustonPutman - I am using MINIO as storage.. I just found this Solr website
You can use the s3.endpoint option to use this BackupRepository with s3-compatible endpoints. Beware that not all s3-compatible endpoints will work with the S3BackupRepository. Minio is an example of an s3-compatible endpoint that does not work with the S3BackupRepository. The S3BackupRepository is only guaranteed to be compatible with AWS S3 and S3Mock. Is this valid with Solr-operator ?

Here are both the YAMLs SOLRCLOUD

`apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
  name: cstone-solr2
spec:
  dataStorage:
    persistent:
      reclaimPolicy: Retain
      pvcTemplate:
        spec:
          resources:
            requests:
              storage: "20Gi"
  replicas: 3
  solrImage:
    repository: myregistry/images/solr
    tag: 8.11.1
    imagePullSecret: "default-gitlab-registry"
    pullPolicy: "Always"

  solrJavaMem: "-Xms1g -Xmx3g"
  customSolrKubeOptions:
    podOptions:
      livenessProbe:
        initialDelaySeconds: 20
        timeoutSeconds: 50
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      readinessProbe:
        initialDelaySeconds: 20
        timeoutSeconds: 50
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3

      podSecurityContext: {}
      #  runAsUser: 1000890000
      resources:
        limits:
          memory: "8G"
        requests:
          cpu: "500m"
          memory: "4G"
  zookeeperRef:
    provided:
      persistence:
        reclaimPolicy: "Retain"
        spec:
          resources:
            requests:
              storage: "20Gi"

  solrOpts: "-Dsolr.autoSoftCommit.maxTime=10000"
  solrGCTune: "-XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8"
  backupRepositories:
    - name: "s3-backups-1"
      s3:
        region: "us-east-1" # Required
        bucket: "cstonebackup" # Required
        credentials:
          accessKeyIdSecret: # Optional
            name: minio-cstonedfs
            key: AWS_ACCESS_KEY_ID
          secretAccessKeySecret: # Optional
            name: minio-cstonedfs
            key: AWS_SECRET_ACCESS_KEY
        endpoint: "http://minio-endpoint-URL:80" # `

    **BACKUP** 
        `apiVersion: solr.apache.org/v1beta1
kind: SolrBackup
metadata:
  name: solr-backup
spec:
  repositoryName: "s3-backups-1"
  solrCloud: cstone-solr2
  location: "cstone-solr2"`
sbbagal13 commented 1 year ago

Hi @HoustonPutman, any suggestion or is there anything wrong with setup

HoustonPutman commented 1 year ago

Is this valid with Solr-operator ?

Yes, the Solr Operator merely sets up your SolrCloud to use the Solr S3 Repository Module, so you will likely not be able to use Minio with the built-in S3 backup feature.

Here are both the YAMLs

I can't imagine why you would be seeing that error though, it's very strange. I would recommend you try with a more recent version of the Solr Operator and see if you still see the error.

sbbagal13 commented 1 year ago

Thanks @HoustonPutman , I suspect this is because of using MINIO as S3 compatible storage. I will try with latest operator as well once.