search

FoundationDB / fdb-kubernetes-operator

A kubernetes operator for FoundationDB
Apache License 2.0
240 stars 83 forks source link

PeerVerificationRule not working #859

Closed tahuy closed 2 years ago

tahuy commented 3 years ago

Hi all, I try to use PeerVerificationRule for TLS, it works for sidecar but not for FDB services Here is my config for certificate

apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: fdb-cluster-certificate namespace: default spec: dnsNames:

  • "*.fdb-cluster.default.svc.cluster.local" secretName: fdb-cluster-certificate issuerRef: name: fdb-cluster-issuer kind: Issuer

Then I apply to my cluster, I use service IP instead of pod IP

apiVersion: apps.foundationdb.org/v1beta1 kind: FoundationDBCluster metadata: name: fdb-cluster spec: version: 6.3.12 faultDomain: key: foundationdb.org/none services: headless: true processCounts: stateless: -1 cluster_controller: 1 storage: 4 log: 4 routing: publicIPSource: "service" headlessService: true mainContainer: enableTls: true peerVerificationRules: "S.subjectAltName<=DNS:.fdb-cluster.default.svc.cluster.local" sidecarContainer: enableTls: true peerVerificationRules: "S.subjectAltName<=DNS:.fdb-cluster.default.svc.cluster.local" processes: general: customParameters:

  • "knob_disable_posix_kernel_aio=1"
  • "locality_test=1" volumeClaimTemplate: spec: resources: requests: storage: "16G" podTemplate: spec: containers:
  • name: foundationdb env:
  • name: FDB_TLS_CERTIFICATE_FILE value: /var/fdb-certs/tls.crt
  • name: FDB_TLS_KEY_FILE value: /var/fdb-certs/tls.key
  • name: FDB_TLS_CA_FILE value: /var/trust-ca/ca.crt volumeMounts:
  • name: fdb-certs mountPath: /var/fdb-certs
  • name: trust-ca mountPath: /var/trust-ca resources: requests: cpu: 250m memory: 128Mi
  • name: foundationdb-kubernetes-sidecar env:
  • name: FDB_TLS_CERTIFICATE_FILE value: /var/fdb-certs/tls.crt
  • name: FDB_TLS_KEY_FILE value: /var/fdb-certs/tls.key
  • name: FDB_TLS_CA_FILE value: /var/trust-ca/ca.crt volumeMounts:
  • name: fdb-certs mountPath: /var/fdb-certs
  • name: trust-ca mountPath: /var/trust-ca resources: requests: cpu: 100m memory: 128Mi limits: cpu: 100m memory: 128Mi initContainers:
  • name: foundationdb-kubernetes-init resources: requests: cpu: 100m memory: 128Mi limits: cpu: 100m memory: 128Mi volumes:
  • name: fdb-certs secret: secretName: fdb-cluster-certificate
  • name: trust-ca secret: secretName: root-secret

And here is error

{"level":"info","ts":1626761979.5837333,"logger":"fdbclient","msg":"Running command","namespace":"default","cluster":"fdb-cluster","path":"/usr/bin/fdb/6.3/fdbcli","args":["/usr/bin/fdb/6.3/fdbcli","--exec","configure new double ssd-2 usable_regions=1 logs=3 proxies=3 resolvers=1 log_routers=-1 remote_logs=-1 regions=[]","-C","/tmp/743120730","--log","--trace_format","xml","--timeout","10","--log-dir","/var/log/fdb"]} {"level":"error","ts":1626761989.6884277,"logger":"fdbclient","msg":"Error from FDB command","namespace":"default","cluster":"fdb-cluster","code":1,"stdout":"\nWARNING: Long delay (Ctrl-C to interrupt)\nSpecified timeout reached -- exiting...\n\nThe database is unavailable; type `status' for more information.\n\n","stderr":"","error":"exit status 1","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/log.(DelegatingLogger).Error\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144\ngithub.com/FoundationDB/fdb-kubernetes-operator/fdbclient.(cliAdminClient).runCommand\n\t/workspace/fdbclient/admin_client.go:187\ngithub.com/FoundationDB/fdb-kubernetes-operator/fdbclient.(cliAdminClient).ConfigureDatabase\n\t/workspace/fdbclient/admin_client.go:224\ngithub.com/FoundationDB/fdb-kubernetes-operator/controllers.UpdateDatabaseConfiguration.Reconcile\n\t/workspace/controllers/update_database_configuration.go:111\ngithub.com/FoundationDB/fdb-kubernetes-operator/controllers.(FoundationDBClusterReconciler).Reconcile\n\t/workspace/controllers/cluster_controller.go:164\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:99"} {"level":"error","ts":1626761989.688877,"logger":"controller","msg":"Error in reconciliation","namespace":"default","cluster":"fdb-cluster","subReconciler":"controllers.UpdateDatabaseConfiguration","requeueAfter":0,"error":"exit status 1","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\ngithub.com/FoundationDB/fdb-kubernetes-operator/controllers.processRequeue\n\t/workspace/controllers/controllers.go:112\ngithub.com/FoundationDB/fdb-kubernetes-operator/controllers.(FoundationDBClusterReconciler).Reconcile\n\t/workspace/controllers/cluster_controller.go:169\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:99"} {"level":"error","ts":1626761989.6890504,"logger":"controller-runtime.manager.controller.foundationdbcluster","msg":"Reconciler error","reconciler group":"apps.foundationdb.org","reconciler kind":"FoundationDBCluster","name":"fdb-cluster","namespace":"default","error":"exit status 1","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:302\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:99"}

johscheuer commented 3 years ago

Can you take a look at the trace files in the container e.g. the operator or the cluster Pods to see if there are any more specific error messages.

johscheuer commented 2 years ago

I'm going to close this stale issue. Feel free to reopen this issue or create a new one.