Using Consul Helm chart in Openshift 4.11 context does not allow to generate system ACL

adaumann commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Currently using Consul Helm within Openshift 4.11.1 context (with Kubernetes 1.24). The automatic generation of system ACL fails

Reproduction Steps

When running helm install with the following values.yml:

consul:
global:
name: consul
datacenter: dc1
image: registry.connect.redhat.com/hashicorp/consul:1.13.1-ubi
imageK8S: registry.connect.redhat.com/hashicorp/consul-k8s-control-plane:0.48.0-ubi
imagePullSecrets:
  - name: 15472453-hcv-redhat-registry-pull-secret
openshift:
  enabled: true
#gossipEncryption:
# autoGenerate: true
tls:
  enabled: true
  enableAutoEncrypt: true
  verify: false
  serverAdditionalDNSSANs:
    - localhost
    - consul-server
    - consul-server.sbk-hcv-infra
    - consul-server.sbk-hcv-infra.svc.cluster.local     
acls:
  manageSystemACLs: true

View error

Logs

Expected behavior

The system ACL with the secret Id is created und Kubernetes secret is build.

adaumann commented 2 years ago

If using the injector same issue:

david-yu commented 2 years ago

Thanks we will take a look into this issue!

kschoche commented 2 years ago

Hi @adaumann - Based on the error message in the last picture it appears as if your helm chart is out of date and behind the consul-k8s binary, can you confirm that you're also using the helm chart 0.48.0 ?

adaumann commented 2 years ago

Hi @adaumann - Based on the error message in the last picture it appears as if your helm chart is out of date and behind the consul-k8s binary, can you confirm that you're also using the helm chart 0.48.0 ?

Yes I'm using 0.48.0 but with the redhat UBI image. The only version I hadd success is 0.43, but only to start an unsecure consul. (No Acl, no TSL, no gossip encryption). See other issue.

david-yu commented 2 years ago

@adaumann How are you deploying OpenShift 4.11 right now? Do you have a link to docs on how to get OpenShift 4.11 up and running quickly?

adaumann commented 2 years ago

Hello, I'm running OpenShift 4.11 as local single-node Kubernetes Cluster in Windows HyperV. In order to run it locally you can use the CRC version for developing. Just register in RedHat and look for "local" and download the installer. Currently there is no storage class defined by default, but you can register a "hostpath" local folder.

install.ps1:

# Install CRC from https://console.redhat.com/openshift/create/local
# Create pull secret from Redhat
# HyperV must be enabled
# Stop IIS, e.g. issreset /stop

crc setup
crc oc-env
crc start -m 20000 -c 8 -p .\pull-secret.txt
Sleep 10
$cr=crc console --credentials -o json | ConvertFrom-Json
$passwd=$cr.clusterConfig.adminCredentials.password

# CRC has normally no storage class for dynamic provision. At hostpath for testing
oc login -u kubeadmin -p $passwd https://api.crc.testing:6443
oc new-project local-path-storage
oc create serviceaccount local-path-provisioner-service-account -n local-path-storage
oc adm policy add-scc-to-user hostaccess -z local-path-provisioner-service-account -n local-path-storage
oc apply -f ./crc-hostpath.yaml
crc status

crc-hostpath.yaml:


#$ Adds storage class as Hostpath for CRC installations
#$ oc login -u kubeadmin -p <passwd> https://api.crc.testing:6443
#$ oc new-project local-path-storage
#$ oc create serviceaccount local-path-provisioner-service-account -n local-path-storage
#$ oc adm policy add-scc-to-user hostaccess -z local-path-provisioner-service-account -n local-path-storage
#$ oc apply -f crc-hostpath.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: local-path-provisioner-role
rules:
- apiGroups: [""]
  resources: ["nodes", "persistentvolumeclaims"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["endpoints", "persistentvolumes", "pods"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: local-path-provisioner-bind
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: local-path-provisioner-role
subjects:
- kind: ServiceAccount
  name: local-path-provisioner-service-account
  namespace: local-path-storage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: local-path-provisioner
  namespace: local-path-storage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: local-path-provisioner
  template:
    metadata:
      labels:
        app: local-path-provisioner
    spec:
      serviceAccountName: local-path-provisioner-service-account
      containers:
      - name: local-path-provisioner
        image: rancher/local-path-provisioner:v0.0.12
        imagePullPolicy: IfNotPresent
        command:
        - local-path-provisioner
        - --debug
        - start
        - --config
        - /etc/config/config.json
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config/
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      volumes:
        - name: config-volume
          configMap:
            name: local-path-config
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: local-path-config
  namespace: local-path-storage
data:
  config.json: |-
        {
                "nodePathMap":[
                {
                        "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
                        "paths":["/mnt/pv-data"]
                }
                ]
        }

darkn3rd commented 2 years ago

Is this the secret you can get with kubectl get --namespace consul secrets/consul-bootstrap-acl-token --template={{.data.token}}? One thing I hit is that if you ever deployed consul service mesh, and didn't delete the namespace, the previous token will be left behind, but this would cause the cluster not to start. I had to delete the namespace to delete any secrets/configs left behind between different tests.

david-yu commented 1 year ago

Will go ahead and close this issue. We've validated validated that this does work with manageSystemACLs = true on OpenShift 4.11 (with consul-k8s 1.0+). We have also addressed the pod security warnings you may encounter with ACLs enabled in Support running with restricted PSA enforcement enabled (part 1) by pglass · Pull Request #2572 · hashicorp/consul-k8s. Please open another issue if you run into any other issues on Consul K8s 1.0.x and above. We will plan to release a patch that includes that PR around mid August.

hashicorp / consul-k8s