Setting imageJolokiaSecret does not enable auth

[toffer]

What did you do? I configured imageJolokiaSecret, but Jolokia started up without auth.

This issue was previously reported in #211, but it was closed before confirming whether or not this was really a problem. I can confirm that this is a bug in need of fixing.

Secret

apiVersion: v1
kind: Secret
metadata:
  name: jolokia-auth
type: Opaque
data:
  username: YWRtaW4=
  password: cGFzc3dvcmQK

CassandraCluster

apiVersion: "db.orange.com/v1alpha1"
kind: "CassandraCluster"
metadata:
  name: cassandra-demo
  labels:
    cluster: k8s.kaas
spec:
  cassandraImage: cassandra:3.11
  bootstrapImage: orangeopensource/cassandra-bootstrap:0.1.5
  configMapName: cassandra-configmap-v1
  dataCapacity: "200Mi"
  dataStorageClass: "standard"
  imagepullpolicy: IfNotPresent
  hardAntiAffinity: false
  deletePVC: true
  autoPilot: false
  gcStdout: false
  autoUpdateSeedList: false
  maxPodUnavailable: 1
  imageJolokiaSecret:
    name: jolokia-auth
  runAsUser: 999
  resources:
    requests:
      cpu: '1'
      memory: 200Mi
    limits:
      cpu: '1'
      memory: 200Mi
  topology:
    dc:
      - name: dc1
        nodesPerRacks: 1
        rack:
          - name: rack1

What did you expect to see?

I expected to see Cassandra start up with this jvm option:

-javaagent:/extra-lib/jolokia-agent.jar=host=0.0.0.0,executor=fixed,authMode=basic,user=admin,password=password

What did you see instead? Under which circumstances?

Cassandra started up with this jvm option:

-javaagent:/extra-lib/jolokia-agent.jar=host=0.0.0.0,executor=fixed

You can see that the Jolokia env vars are set in the cassandra container:

$ kubectl -n cassandra exec -it cassandra-demo-dc1-rack1-0 -c cassandra -- /bin/bash
cassandra@cassandra-demo-dc1-rack1-0:/$ env | sort | grep JOLOKIA
CASSANDRA_AUTH_JOLOKIA=true
JOLOKIA_PASSWORD=password
JOLOKIA_USER=admin

However, these env vars are not set in the bootstrap container, which is where run.sh looks for them in order to update the jolokia jvm option. If you add a sleep 3600 to post_run.sh and exec into the bootstrap pod, you can see that the JOLOKIA env vars are missing:

$ kubectl -n cassandra exec -it cassandra-demo-dc1-rack1-0 -c bootstrap -- /bin/bash
cassandra@cassandra-demo-dc1-rack1-0:/$ env | sort | grep JOLOKIA
JOLOKIA_VERSION=1.6.1

Bootstrap env vars are set in cassandracluster.bootstrapContainerEnvVar(), but the Jolokia env vars are not included in the list of env vars set. https://github.com/Orange-OpenSource/casskop/blob/9253cd6115accbd715007817cc4b78b0c2865866/pkg/controller/cassandracluster/generator.go#L577

The Jolokia env vars are added to the cassandra container (and only the cassandra container) here: https://github.com/Orange-OpenSource/casskop/blob/9253cd6115accbd715007817cc4b78b0c2865866/pkg/controller/cassandracluster/generator.go#L381-L408

Environment

• casskop version:

v0.5.6-release

• Kubernetes version information:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T21:51:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-14T07:30:52Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  $ kind version
  kind v0.9.0 go1.15.2 darwin/amd64

• Cassandra version:

3.11.8

Possible Solution

For the Jolokia credentials in particular, I'd like the JOLOKIA_USER and JOLOKIA_PASSWORD env vars to be set in both the bootstrap and cassandra containers.

In general, I don't see any reason why the env vars in the bootstrap and cassandra containers should differ. Is there a reason why they shouldn't be the same?

[cscetbon]

Correct, that's what should happen at https://github.com/Orange-OpenSource/casskop/blob/master/docker/bootstrap/files/run.sh#L182-L207

In general, I don't see any reason why the env vars in the bootstrap and cassandra containers should differ. Is there a reason why they shouldn't be the same?

Yes, I made that change. The reason is that the configuration is generated by the bootstrap init-container. The cassandra container just uses it so need a few if not no variables. I think even https://github.com/Orange-OpenSource/casskop/blob/master/pkg/controller/cassandracluster/generator.go#L382-L406 is not needed as you can see at https://github.com/Orange-OpenSource/casskop/blob/master/docker/bootstrap/bootstrap.sh.

If the authentication is not set I need to take a look at it.

[toffer]

In the case of JOLOKIA_USER and JOLOKIA_PASSWORD, it would definitely be helpful to have the env vars set in both the bootstrap container and in the cassandra container. As discussed above, the bootstrap container needs the env vars so that it can update the jolokia jvm option in run.sh.

But, the JOLOKIA env vars are also helpful in the cassandra container. For example, the readiness-probe.sh script hits the jolokia endpoint. Since we're using auth, I've updated our readiness-probe.sh to work both with and without credentials. It starts like:

# Add Jolokia credentials, if defined in environment.
if [[ -z $JOLOKIA_USER ]] || [[ -z $JOLOKIA_PASSWORD ]]; then
    USER_OPT=""
else
    USER_OPT="--user $JOLOKIA_USER:$JOLOKIA_PASSWORD"
fi

# We check when the node is up and in normal state
CURL="/opt/bin/curl $USER_OPT -s --connect-timeout 0.5"
BASE_CMD="http://$POD_IP:8778/jolokia/read/org.apache.cassandra.db:type=StorageService"

[toffer]

If you like, I can put together a PR that moves where the JOLOKIA env vars get added, so that the addition happens in bootstrapContainerEnvVar(). That way, the JOLOKIA env vars will be present on both the bootstrap and cassandra containers.

And, I can update readiness-probe.sh as above to read the credentials from the env vars and use them.

Let me know if you'd like me to proceed.

[cscetbon]

@toffer yeah I forgot about the readiness probe that uses those variables.

And, I can update readiness-probe.sh as above to read the credentials from the env vars and use them.

Wwhat you mean ? it already uses env vars as shown in the script you pasted

[toffer]

The current version doesn't use those env vars: https://github.com/Orange-OpenSource/casskop/blob/master/docker/bootstrap/files/readiness-probe.sh#L17-L24

I was suggesting that I would update the current version of readiness-probe.sh to work like the example I pasted above.

[cscetbon]

Oh I see, I thought it did. Then 👍