Flexible user password initialization

[gregwebs]

Problem

Our current initializer job that sets the user password suffers from a problem:

• It puts SQL into a secret with a password. When we retrieve the password in an automated way, we must run an awk command: it is a very fragile situation

Both the initializer job and other passwords (backup) have passwords passed through helm.

• This is insecure unless the user stores their helm config as a secret (not the default)
• This can present its own escaping issues

Solving Escaping

The initializer job is structured the way it is to avoid escaping issues. However, we have determined two alternatives:

• Use bash and mysqladmin
• Use a docker image that has a mysql library installed.
  • We can use a dynamic language like Python to avoid a compilation step.
  • It should also be possible to use a pre-compiled go program, but then we would need to distribute our own image
• Secrets are given to the pod as either a file or an environment variable that can be read in by the program

Avoiding helm

All passwords should be created as secrets by the user. The manifests just reference the secret names.

[gregwebs]

Also, gcp.credentialsData in the backup chart should be created as a secret.

[tennix]

With py-mysqldb installed on alpine linux image and the initial root password stored in ROOT_PASSWORD environment, the root password can be set with the following command:

python -c "import os, MySQLdb; MySQLdb.connect(host='127.0.0.1', port=4000, user='root').cursor().execute(\"set password for 'root'@'%%' = %s; flush privileges;\", (os.environ.get('ROOT_PASSWORD'),))"

There is already a Python mysqlclient image on DockerHub: https://github.com/tnir/mysqlclient.

As for your suggestion, I prefer to make the secrets optional: If users provide secret names for the passwords, then helm would not create these secrets. But users can also set passwords and credentials in the values.yaml and let helm to create these secrets.

[gregwebs]

I prefer to make the secrets optional: If users provide secret names for the passwords, then helm would not create these secrets. But users can also set passwords and credentials in the values.yaml and let helm to create these secrets.

There is a down-side to maintaining two ways of doing things, particularly where one is insecure by default. Supporting insecure by default techniques means we must spend time documenting how to do things in a secure way.

The insecure by default comes in two forms

• Helm will store this its chart data in an ordinary ConfigMap by default. There is a configuration option to store the entire configuration as a secret, but that still means that giving access to helm is equivalent to giving access to all the secrets.
• Putting a secret into a chart essentially means it will be checked into version control, which is a poor place for secrets. You need to have a process to -set the secret at the command line.

[tennix]

Either way we have to document how to do things in a secure way. And the secure way is complicated for new users. Using insecure way we can allow new users quickly get TiDB up and running. If advanced users want the secure way, they can find the method in the document.

[gregwebs]

I agree the insecure way is easy for a demo that you are going to destroy, but it is actually more complicated once you realize you have to save your password somewhere.

The secure way just requires running kubectl create secret generic password --from-literal=key=value. Its quite easy even if more verbose than adding -set tidb.password=value to your helm install.

[tennix]

Yes, creating it may be easy, but the secret is outside of the helm control. Users may forget about that. Besides k8s secrets are not always encrypted. There is lots of work to do to keep the credential data real safe. So to keep it really secure, we must add more document about this for example how to enable k8s encryption at rest.

[gregwebs]

Yes, creating it may be easy, but the secret is outside of the helm control. Users may forget about that.

Good point. Note though that if the tidb cluster is installed into a new namespace, then deleting the namespace will delete the secret.

Besides k8s secrets are not always encrypted. There is lots of work to do to keep the credential data real safe. So to keep it really secure, we must add more document about this for example how to enable k8s encryption at rest.

I think we can just rely on the existing Kubernetes secret documentation. Whereas if we put secret values through helm they must understand both K8s secrets and all of the issues of having secrets in helm.

[gregwebs]

For managing the secret lifecycle, I think we can still delete the secret automatically (perhaps adding a helm delete hook somewhere).

[gregwebs]

Related to refactoring secrets, we currently generate a random password if none is given. However, that workflow may be broken: https://github.com/helm/helm/issues/3053

[gregwebs]

I have an init job now that looks like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{.Values.ClusterName}}-tidb-initializer
  labels:
    app.kubernetes.io/instance: demo 
    app.kubernetes.io/component: tidb-initializer
spec:
  backoffLimit: 1000
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: demo 
        app.kubernetes.io/component: tidb-initializer
    spec:
      restartPolicy: OnFailure
      containers:
      - name: mysql-client
        image: tnir/mysqlclient
        imagePullPolicy: "IfNotPresent"
        command:
        - python
        - -c
        - |
          import os, MySQLdb
          sql="""
          SET PASSWORD FOR 'root'@'%%' = '%s';
          FLUSH PRIVILEGES;
          """ % (os.environ.get('ROOT_PASSWORD'))
          # import sys; sys.stdout.flush(); print(sql)                                                                                                                                         
          conn = MySQLdb.connect(host='127.0.0.1', port=4000, user='root')
          for line in sql.splitlines():
              if line == '': 
                  continue
              conn.cursor().execute(line)
              conn.commit()
        env:
        - name: ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: "{{ .Values.tidb.passwordSecretName }}"
              key: root 

[tennix]

Related to refactoring secrets, we currently generate a random password if none is given. However, that workflow may be broken: helm/helm#3053

Yes, this is the biggest drawback of current approach. I agree with you to let user create secret outside of helm and reference the secret in values.yaml.