Support configuring certs when backup and restore using remote storage with TLS enabled

DanielZhangQD commented 4 years ago

Feature Request

Describe your feature request related problem:

In https://github.com/pingcap/tidb-operator/issues/2277, the user reports the issue that if TLS is enabled for remote storage, such as MinIO, the backup will fail.

Describe the feature you'd like:

Support configuring certs when backup and restore using remote storage with TLS enabled.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

kennytm commented 4 years ago

Both TiKV and BR need to connect to the S3 service.

On the TiKV side, we are using HttpClient::new_with_config in rusoto_util, so self-signed certificates won't be accepted. The client must be initiated using a custom connector.

On the BR side, we could override the default HTTPClient.

How the TLS certificates should be shared? There are two solutions, either

automatically trust the self-signed certs specified in the [security] configs, or
allow user to provide the TLS certs only for S3 (--s3.ca, --s3.cert, --s3.key), and send these cert files via gRPC to TiKV.

DanielZhangQD commented 4 years ago

How the TLS certificates should be shared? There are two solutions, either

automatically trust the self-signed certs specified in the [security] configs, or allow user to provide the TLS certs only for S3 (--s3.ca, --s3.cert, --s3.key), and send these cert files via gRPC to TiKV.

I am not quite sure about this, @kolbe could you please help confirm Kenny's question?

kolbe commented 4 years ago

I'm not really sure what the question is here.

If the question is "how to share certificates", I think the answer is to use k8s secrets if this is running in tidb-operator.

If you're trying to figure out how to share secrets from BR to TiKV when not running in tidb-operator, I think it's pretty straightforward to ship them from BR to TiKV as part of a one-off API call, so you must be asking about something else?

I don't understand the either/or between providing TLS certs using configuration options and the item about trusting self-signed certs. Can you please clarify that?

DanielZhangQD commented 4 years ago

I'm not really sure what the question is here.

Users request BR & TiKV to support backup to the MinIO (S3 compatible) storage that with TLS enabled as described in pingcap/tidb-operator#2277.

kolbe commented 4 years ago

There seems to be some confusion here even about the concept of a "self-signed certificate". These certificates are not self-signed, they are just signed by a CA that is not a part of the default trust store used by the OS or whatever TLS library is used.

So, is the issue here just how to communicate CA trust information to the BR components? If running in Kubernetes, this should perhaps be done using Kubernetes "secrets" to share the CA certificate itself and a command-line flag to point to the mounted certificate file. That's the approach used by other components.

The software should absolutely not "automatically" trust any certificates. Certificate trust should always be established by explicitly adding a CA certificate to the trust chain used by the TLS library.

Does that help? Am I still missing the question being posed?

Smana commented 3 years ago

Hi @kolbe , actually we're facing the same problem using the tidb-initializer resource: When we run the job it uses a mysql client and returns the following error:

Traceback (most recent call last):
  File "/usr/local/bin/start_script.py", line 5, in <module>
    conn = MySQLdb.connect(host=host, port=port, user='root', connect_timeout=5, ssl={'ca': '/var/lib/tidb-client-tls/ca.crt', 'cert': '/var/lib/tidb-client-tls/tls.crt', 'key': '/var/lib/tidb-client-tls/tls.key'})
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/__init__.py", line 84, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 179, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (2026, 'SSL connection error: unable to get issuer certificate')

Because the cert-manager doesn't include the root CA. Please check this issue https://github.com/jetstack/cert-manager/issues/2166

So ideally we need to be able to define the CA path on all the objects. I'll have a look on the operator to see if I can propose something.

pingcap / br