sathyu commented 5 years ago

Hello, I am deploying this in k8 and getting following errors, any idea what that I am missing?
I am deploying this as non-admin user.

$ kubectl logs stolon-sentinel-7754964b89-8vmv4 2019-01-29T21:22:26.067Z INFO cmd/sentinel.go:1962 sentinel uid {"uid": "dfa105e8"} 2019-01-29T21:22:26.071Z INFO cmd/sentinel.go:80 Trying to acquire sentinels leadership ERROR: logging before flag.Parse: I0129 21:22:26.071509 1 leaderelection.go:174] attempting to acquire leader lease... ERROR: logging before flag.Parse: E0129 21:22:26.146967 1 leaderelection.go:224] error retrieving resource lock k8poc-sathya/stolon-cluster-kube-stolon: configmaps "stolon-cluster-kube-stolon" is forbidden: User "system:serviceaccount:k8poc-sathya:default" cannot get configmaps in the namespace "k8poc-sathya"

lwolf commented 5 years ago

Hi, as log record says

User "system:serviceaccount:k8poc-sathya:default" cannot get configmaps in the namespace "k8poc-sathya"

the user does not have permissions to get configmaps. You need to check ServiceAccount/Role/RoleBinding to make sure that the user has required permissions.

sathyu commented 5 years ago

Hello , As understanding, I can do all admin works within my name space. and I was told to user my service account which I am doing. Still getting error.
So, question 1. we I need access to kube-system namespace, if so , is just "Read" sufficient.

if I don't get access to kube-system what is the way to implement this ?
1. Are you available for 1:1 talk , I am ready to compensate your time (serious).

$ Error from server (Forbidden): deployments.extensions is forbidden: User "system:serviceaccount:k8poc-sathya:k8-poc-sathya" cannot list deployments.extensions in the namespace "kube-system"

lwolf commented 5 years ago

It shouldn't require access to the kube-system namespace at all. Could you please provide more information about your setup, will see what I can do?

what k8s version do you use?
what stolon version do you use?
what namespace did you install the chart?
could you post content of your values.yaml stripping out (if any) sensitive information

sathyu commented 5 years ago

Step1: $ kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Step 2:
imageTag: "v0.12.0-pg10" Namespace = k8poc-sathya Step 3: $ helm install --name k8poc-sathya -f values.yaml . Error: pods is forbidden: User "system:serviceaccount:k8poc-sathya:k8-poc-sathya" cannot list pods in the namespace "kube-system"

step 4: values.yaml

Hello. Below is the values.yaml entries: $ cat values.yaml

# Default values for Stolon Helm Chart.
# This is a YAML-formatted file.
## Declare variables to be passed into your templates.

## Override the name of the Chart.
##
# nameOverride:

## Stolon image.
##

image: "sorintlab/stolon"

## Stolon image version.
## ref: https://hub.docker.com/r/sorintlab/stolon/tags/
##
imageTag: "v0.12.0-pg10"

## Specify a imagePullPolicy: 'Always' if imageTag is 'latest', else set to 'IfNotPresent'.
## ref: https://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
# imagePullPolicy:

## Configuration values for Stolon.

# Set custom stolon cluster name
clusterName: "kube-stolon"
debug: false

## log slow queries
# disabled by default
slow_queries:
  enabled: false
  min_duration: 300

ports:
  internalPort: 5432
  externalPort: 5432

store:
  ##  Backend could be one of the following:
  ## - etcdv2
  ## - etcdv3
  ## - consul (should work, but not tested yet)
  ## - kubernetes (should work, but not tested yet)
  backend: kubernetes
  ## store endpoints MUST be set for etcd/consul backends
  #  endpoints: "http://etcd-etcd-0.etcd-etcd:2379,http://etcd-etcd-1.etcd-etcd:2379,http://etcd-etcd-2.etcd-etcd:2379"

pgReplUsername: "repluser"
## set password for the repluser
## default is 40 random chars
pgReplPassword: "replPassword"

pgSuperuserName: "stolon"

## set password for the superuser
## default is 40 random chars
pgSuperuserPassword: "stolon123"

sentinel:
  replicas: 3

  ## Configure resource requests and limits.
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ##

  resources:
    requests:
      cpu: "100m"
      memory: "512Mi"

  ## Configure nodeSelector, tolerations and affinity.
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node
  ##

  affinity: {}
  nodeSelector: {}
  tolerations: []

proxy:
  replicas: 2
  ## Set serviceType to nodePort if needed
  ## proxy is used to route RW requests to the master
  # serviceType: NodePort

  ## Configure resource requests and limits.
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ##

  resources:
    requests:
      cpu: "100m"
      memory: "512Mi"

  ## Configure nodeSelector, tolerations and affinity.
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node
  ##

  affinity: {}
  nodeSelector: {}
  tolerations: []

keeper:
  replicas: 2
  ## Set serviceType to nodePort if needed
  ## keeper service is used to route RO requests to all nodes
  # serviceType: NodePort

  ## configure ssl for client access
  # create certificates according to these instructions: https://www.postgresql.org/docs/9.6/static/ssl-tcp.html
  # to enable encrypted traffic, servert.crt and server.key are required, by that name.
  # the use of ** Client Certificates ** is not supported
  client_ssl:
    enabled: false
    certs_secret_name: pg-cert-secret

  ## Configure resource requests and limits.
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ##

  resources:
    requests:
      cpu: "100m"
      memory: "512Mi"

  ## Configure nodeSelector, tolerations and affinity.
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node
  ##

  affinity: {}
  nodeSelector: {}
  tolerations: []
## Persistent Volume Storage configuration.
## ref: https://kubernetes.io/docs/user-guide/persistent-volumes
##

persistence:
  ## Enable persistence using Persistent Volume Claims.
  ##
  enabled: false

  ## Persistent Volume Access Mode.
  ##
  accessMode: ReadWriteOnce

  ## Persistant Volume Storage Class Name
  storageClassName: standard

  ## Persistent Volume Storage Size.
  ##
  size: 25Gi

rbac:
  # Specifies whether RBAC resources should be created
  create: true

serviceAccount:
  # Specifies whether a ServiceAccount should be created
  create: true
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name: k8-poc-sathya

lwolf commented 5 years ago

I've just deployed this chart with the values.yaml file you've pasted. I don't have k8s 1.10 so I tested on my 1.12.3. Everything works fine.

$ kubectl get pods
k8poc-sathya-stolon-keeper-0                                      1/1       Running     0          1m
k8poc-sathya-stolon-keeper-1                                      1/1       Running     0          46s
k8poc-sathya-stolon-lz6ws                                         0/1       Completed   0          1m
k8poc-sathya-stolon-proxy-64dfb7b59-7jmx8                         1/1       Running     0          1m
k8poc-sathya-stolon-proxy-64dfb7b59-vnt7w                         1/1       Running     0          1m
k8poc-sathya-stolon-sentinel-59dc875688-2gdxw                     1/1       Running     0          1m
k8poc-sathya-stolon-sentinel-59dc875688-97r6d                     1/1       Running     0          1m
k8poc-sathya-stolon-sentinel-59dc875688-rmrpr                     1/1       Running     0          1m

Did you try reinstalling the chart from scratch? Did helm install actually succeeded? Did you previously install anything using helm? maybe it's misconfigured

sathyu commented 5 years ago

Hello again, Now stolon-sentinel is up and running but logged below error at pod level: 2019-02-06T19:02:20.069Z INFO cmd/sentinel.go:1962 sentinel uid {"uid": "70089ff6"} 2019-02-06T19:02:20.145Z INFO cmd/sentinel.go:80 Trying to acquire sentinels leadership ERROR: logging before flag.Parse: I0206 19:02:20.145433 1 leaderelection.go:174] attempting to acquire leader lease... ERROR: logging before flag.Parse: E0206 19:02:20.152184 1 leaderelection.go:224] error retrieving resource lock k8poc-sathya/stolon-cluster-kube-stolon : configmaps "stolon-cluster-kube-stolon" is forbidden: User "system:serviceaccount:k8poc-sathya:stolon-sa" cannot get configmaps in the namespace "k8poc-sat hya" 2019-02-06T19:02:20.162Z ERROR cmd/sentinel.go:1815 error retrieving cluster data {"error": "failed to get latest version of configmap: configm aps \"stolon-cluster-kube-stolon\" is forbidden: User \"system:serviceaccount:k8poc-sathya:stolon-sa\" cannot get configmaps in the namespace \"k8poc-sathya\ ""} AND stolok-keeper is crashlooping with error: 2019-02-06T19:20:34.151Z FATAL cmd/keeper.go:117 cannot get current user: cannot detect current user.

For me, role.yaml (as provided ) did not work , it errors out. So I changed it like below and ran fine. kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: k8poc-sathya name: stolon rules:

apiGroups: ["extensions", "apps"] resources: ["deployments", "replicasets"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] Role-binding: apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: rb-stolon subjects:
kind: ServiceAccount name: stolon-sa namespace: k8poc-sathya roleRef: kind: Role name: stolon apiGroup: rbac.authorization.k8s.io

As always , your help is appreciated.

lwolf commented 5 years ago

so, it seems that you resolved your problem and the issue could be closed?

I'll keep in mind that additional roles might be required. But for now I can't reproduce it. Official stolon example does not have this as well - https://github.com/sorintlab/stolon/blob/master/examples/kubernetes/role.yaml

sathyu commented 5 years ago

Hello I did resolved the error I was getting during helm install. Now, I am getting below error when starting stolon-keeper pods. To again emphasize, I have to run this as cluster non-admin user.
I have made some changes to keeper.yaml like below

chown stolon:stolon $STOLON_DATA

        exec stolon-keeper --data-dir $STOLON_DATA
        #exec gosu stolon stolon-keeper --data-dir $STOLON_DATA

Both chown and gosu will not work for me, errors out. let me know how can I fix this error. Thanks "cannot get current user: cannot detect current user"

lwolf / stolon-chart

User cannot get configmaps in the namespace #27

chown stolon:stolon $STOLON_DATA