Open tfulcrand opened 4 years ago
@tfulcrand Please upgrade TiDB Operator to v1.1.0-rc.2.p1
and also set tidbBackupManagerImage: pingcap/tidb-backup-manager:v1.1.0-rc.2.p1
and try again.
Please post the pod logs and the yaml of backup including the status
section here if fails again.
@DanielZhangQD As you asked, we have upgraded TiDB operator and TiDB backup manager version and now we see more details on the previous errors. We get this kind of errror : x509: certificate signed by unknown authority
.
In fact we use cert-manager and vault-pki to automatically manage certificates and then we have an private CA which is no trusted by the backup manager. To temporally workaround this problem we have added a LetsEncrypt wildecard certificate for all subdomains in kube.dm.gg but it's not recommended for security. (BTW with this workaround backups succedded on minio)
Ideally we would like to be able to import our CA to backup manager, do you plan to add this feature ?
@tfulcrand tidb-backup-manager already supports TLS, please follow the doc to create ${cluster_name}-tidb-client-secret
and follow the doc to create ${cluster_name}-cluster-client-secret
in the target namespace of the tidb cluster.
tidb-backup-manager will mount those two secrets automatically to connect to TiDB, PD, and TiKV.
You can also create a private secret with the tidb client certs and configure the secret name to the spec.from.tlsClient.tlsSecret
and backup-manager will use this secret instead of ${cluster_name}-tidb-client-secret
. cc @weekface
@tfulcrand With v1.1.0 TiDB Operator, the separate tidb client cert for Backup CR is changed to spec.from. tlsClientSecretName
, you can refer to https://pingcap.com/docs/tidb-in-kubernetes/stable/enable-tls-for-mysql-client/#using-cert-manager step 5 for its creation.
@DanielZhangQD Thanks for your replies.
But The problem is still present... Probably the explanation is not clear.
We setup a Minio instance as backup storage. This Minio instance stands behind an Ingress component with TLS enable. The certificate uses by the Ingress is generated by cert-manager and vault-pki (for auto management). With this configuration when lauching a CRD Backup object, the log of associated pod shows this message : x509: certificate signed by unknown authority
and backup task fails.
Currently we use a wildcard LetsEncrypt certificate in the Ingress as workaround but we would like to use our generated certificate in the future.
@tfulcrand Sorry for the late response, so basically, the BR and TiKV should support loading customized certs and connects to the MinIO instance with https enabled, right?
@tfulcrand would you please explain more about the request to meet your requirement?
@DanielZhangQD I'll try to explain too: We already tried your documentation regarding cert-manager usage. In this documentation you make use of a self-signed certificate issued by cert-manager itself. That means that the CA is created by cert-manager.
For our use-case we use cert-manager but with a vault pki issuer. In that case the CA is generated on Vault side and we use this CA on all our internal services within kubernetes. That means that, at some point, this CA must be trusted between 2 services in order to communicate with each other using TLS.
So here we'd like to configure the backup job to trust our CA for the MinIO communication.
The same way as you will do a curl --cacert cacert.pem https://minio_server/
for a curl command :)
I hope this requirement is clear enough for you, don't hesitate if you need any detail.
I think cert-manager could generate certs with an user-defined CA, could you please help confirm this? @weekface
@Smana Have you tried the procedures in this doc? If the vault-issuer is created, it should be good to follow the rest of our docs to create certs for the components.
This is more about how to handle self signed certificate on client side @DanielZhangQD . There's no way to set a secure trusted connection from the backup job (here the client initiating the TLS connection) to the Minio instance using our internal private CA without explicitly trusting the CA on client side.
So my understanding here https://github.com/pingcap/tidb-operator/issues/2277#issuecomment-647322439 is correct, right? The backup job and TiKV should support configuring certs that can be used to connect to 3rd party services such as Minio. But for now, they can only load the certs to be used internally among the TiDB cluster components.
Hi @DanielZhangQD , yes your comment is right :) loading a certificate, ca cert would be nice.
OK, https://github.com/pingcap/br/issues/409 is created for BR.
Bug Report
What version of Kubernetes are you using? Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.9", GitCommit:"2e808b7cb054ee242b68e62455323aa783991f03", GitTreeState:"clean", BuildDate:"2020-01-18T23:24:23Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
What version of TiDB Operator are you using? TiDB Operator Version: version.Info{GitVersion:"v1.1.0-rc.2", GitCommit:"6f42736bd0f33aed65903058f43a9005e806680a", GitTreeState:"clean", BuildDate:"2020-04-16T01:23:10Z", GoVersion:"go1.13", Compiler:"gc", Platform:"linux/amd64"}
What's the status of the TiDB cluster pods?
What did you do?
We define a CRD Backup object as follow :
What did you expect to see? Backup piece present on our Minio instance.
What did you see instead? Minio instance stays empty and we can see in bakup pod logs this :