Sjnahak commented 4 years ago

Hi Team, Thanks for updating the repo with adapt to Helm version 3:

I was testing gerrit master setup yesterday , please find setup details

Helm Version : 3
Kuberenetes version 1.18.0
OS: Rhel
Kubeadm cluster  with 3 master and 2 worker node
Ingress Deployed as Daemonsets

As Per Prerequisites

Ingress /Nfs and helm was required , since i had nfs server already created/provided by nfs team created nfs client provisioner using

[root@k8smaster nfs]# helm install stable/nfs-client-provisioner --set nfs.server=xxxxxx --set nfs.path=/export path --generate-name
OUTPUT
NAME: nfs-client-provisioner-1593515904
LAST DEPLOYED: Tue Jun 30 11:18:26 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

[root@k8smaster nfs]# kubectl get storageclass
NAME         PROVISIONER                                       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client   cluster.local/nfs-client-provisioner-1593515904   Delete          Immediate           true                   5m28s

I have passed these nfs details into values.yaml But deployment fails with below

Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "gerrit-gerrit-stateful-set-0": pod has unbound immediate PersistentVolumeClaims

There were two storage class defined in values.yaml

 First i tried with storage class nfs  share and default with provisioner kubernetes.io/aws-ebs  received same error  regarding volume ( (i was sure this will fail because we don't have aws)

Second tried with passing nfs share at both places which created as Prerequisites

Values.yaml Used by me

images:
  registry:
    # The registry name must NOT contain a trailing slash
    name:
    ImagePullSecret:
      # Leave blank, if no ImagePullSecret is needed.
      name: image-pull-secret
      # If set to false, the gerrit chart expects either a ImagePullSecret
      # with the name configured above to be present on the cluster or that no
      # credentials are needed.
      create: false
      username:
      password:
  version: latest
  imagePullPolicy: Always

storageClasses:
  # Storage class used for storing logs and other pod-specific persisted data
  default:
    # If create is set to false, an existing StorageClass with the given
    # name is expected to exist in the cluster. Setting create to true will
    # create a storage class with the parameters given below.
    name: nfs-client
    create: false
    provisioner: nfs-client-provisioner-1593515904
    reclaimPolicy: Delete
    # Use the parameters key to set all parameters needed for the provisioner
    parameters:
      mountOptions: vers=4.1

  # Storage class used for storing git repositories. Has to provide RWM access.
  shared:
    # If create is set to false, an existing StorageClass with RWM access
    # mode and the given name has to be provided.
    name: nfs-client
    create: false
    provisioner: nfs-client-provisioner-1593515904
    reclaimPolicy: Delete
    # Use the parameters key to set all parameters needed for the provisioner
    parameters:
      mountOptions: vers=4.1

gitRepositoryStorage:
  externalPVC:
    use: false
    name: git-filesystem-pvc
  size: 3Gi

gitGC:
  image: k8sgerrit/git-gc

  schedule: 0 6,18 * * *

  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 100m
      memory: 256Mi

  logging:
    persistence:
      enabled: true
      size: 1Gi

gerrit:
  images:
    gerritInit: k8sgerrit/gerrit-init
    gerrit: k8sgerrit/gerrit

  replicas: 1
  updatePartition: 0

  # The memory limit has to be higher than the configures heap-size for Java!
  resources:
    requests:
      cpu: 1
      memory: 3Gi
    limits:
      cpu: 1
      memory: 4Gi

  persistence:
    enabled: true
    size: 3Gi

  livenessProbe:
    initialDelaySeconds: 30
    periodSeconds: 5

  readinessProbe:
    initialDelaySeconds: 5
    periodSeconds: 1

  service:
    type: NodePort
    http:
      port: 80

  ingress:
    host: helm-gerritci.example.com
    # The maximum body size to allow for requests. Use "0" to allow unlimited
    # reuqest body sizes.
    maxBodySize: 50m
    additionalAnnotations: {}
    #  nginx.ingress.kubernetes.io/server-alias: example.com
    #  nginx.ingress.kubernetes.io/whitelist-source-range: xxx.xxx.xxx.xxx
    tls:
      enabled: false
      cert: |-
        -----BEGIN CERTIFICATE-----

        -----END CERTIFICATE-----
      key: |-
        -----BEGIN RSA PRIVATE KEY-----

        -----END RSA PRIVATE KEY-----

  # `gerrit.keystore` expects a base64-encoded Java-keystore
  # Since Java keystores are binary files, adding the unencoded content and
  # automatic encoding using helm does not work here.
  keystore:

  plugins:
    packaged:
    - commit-message-length-validator
    - download-commands
    - replication
    - reviewnotes
    downloaded:
    # - name: delete-project
    #   url: https://example.com/gerrit-plugins/delete-project.jar
    #   sha1:

    # Only downloaded plugins will be cached. This will be ignored, if no plugins
    # are downloaded.
    cache:
      enabled: false
      size: 1Gi

  config:
    # Some values are expected to have a specific value for the deployment installed
    # by this chart to work. These are marked with `# FIXED`.
    # Do not change them!
    gerrit: |-
      [gerrit]
        basePath = git # FIXED
        serverId = gerrit-1
        # The canonical web URL has to be set to the Ingress host, if an Ingress
        # is used. If a LoadBalancer-service is used, this should be set to the
        # LoadBalancer's external IP. This can only be done manually after installing
        # the chart, when you know the external IP the LoadBalancer got from the
        # cluster.
        canonicalWebUrl = http://helm-gerritci.example.com/
        disableReverseDnsLookup = true
      [index]
        type = LUCENE
      [auth]
        type = DEVELOPMENT_BECOME_ANY_ACCOUNT
      [httpd]
        # If using an ingress use proxy-http or proxy-https
        listenUrl = proxy-http://*:8080/
      [sshd]
        listenAddress = off
      [transfer]
        timeout = 120 s
      [user]
        name = Gerrit Code Review
        email = gerrit@example.com
        anonymousCoward = Unnamed User
      [cache]
        directory = cache
      [container]
        user = gerrit # FIXED
        javaHome = /usr/lib/jvm/java-1.8-openjdk # FIXED
        javaOptions = -Djavax.net.ssl.trustStore=/var/gerrit/etc/keystore # FIXED
        javaOptions = -Xms200m
        # Has to be lower than 'gerrit.resources.limits.memory'. Also
        # consider memories used by other applications in the container.
        javaOptions = -Xmx2g

    secure: |-
      # Password for the keystore added as value for 'gerritReplica.keystore'
      # Only needed, if SSL is enabled.
      #[httpd]
      #  sslKeyPassword = gerrit

      # Credentials for replication targets
      # [remote "replica"]
      # username = git
      # password = secret

    replication: |-
      [gerrit]
        autoReload = false
        replicateOnStartup = true
        defaultForceUpdate = true

      # [remote "replica"]
      # url = http://gerrit-replica.example.com/git/${name}.git
      # replicationDelay = 0
      # timeout = 30

Deployment Step

root@k8smaster gerrit]# helm install gerrit . -f values.yaml
NAME: gerrit
LAST DEPLOYED: Tue Jun 30 14:26:06 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
A primary Gerrit instance has been deployed.
==================================

Gerrit may be accessed under: helm-gerritci.example.com

Describe pod Output

[root@k8smaster gerrit]# kubectl describe pod gerrit-gerrit-stateful-set-0
Name:           gerrit-gerrit-stateful-set-0
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=gerrit
                chart=gerrit-0.2.0
                controller-revision-hash=gerrit-gerrit-stateful-set-6b46cb67fc
                heritage=Helm
                release=gerrit
                statefulset.kubernetes.io/pod-name=gerrit-gerrit-stateful-set-0
Annotations:    chartRevision: 1
Status:         Pending
IP:
IPs:            <none>
Controlled By:  StatefulSet/gerrit-gerrit-stateful-set
Init Containers:
  site-cleanup:
    Image:      busybox
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -c
    Args:
      # Remove directories that should be mounted rather than exist with the
      # rest of the site
      [ ! -L /var/gerrit/git ] && rm -rf /var/gerrit/git
      rm -f /var/gerrit/logs/gerrit.pid

    Environment:  <none>
    Mounts:
      /var/gerrit from gerrit-site (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pxktr (ro)
  gerrit-init:
    Image:      k8sgerrit/gerrit-init:latest
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/ash
      -ce
    Args:
      symlink_config_to_site(){

        ln -sf /var/config/gerrit.config /var/gerrit/etc/gerrit.config
        ln -sf /var/config/secure.config /var/gerrit/etc/secure.config
      }

      mkdir -p /var/gerrit/etc
      symlink_config_to_site

      if [ ! -d /var/gerrit/git ]; then
        ln -sf /var/mnt/git /var/gerrit/
      fi

      /var/tools/gerrit_init.py \
        -c /var/config/gerrit-init.yaml \
        -s /var/gerrit

      symlink_config_to_site

    Environment:  <none>
    Mounts:
      /var/config/gerrit-init.yaml from gerrit-init-config (rw,path="gerrit-init.yaml")
      /var/config/gerrit.config from gerrit-config (rw,path="gerrit.config")
      /var/config/secure.config from gerrit-secure-config (rw,path="secure.config")
      /var/gerrit from gerrit-site (rw)
      /var/mnt/git from git-filesystem (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pxktr (ro)
Containers:
  gerrit:
    Image:      k8sgerrit/gerrit:latest
    Port:       8080/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     1
      memory:  4Gi
    Requests:
      cpu:        1
      memory:     3Gi
    Liveness:     http-get http://:gerrit-port/config/server/version delay=30s timeout=1s period=5s #success=1 #failure=3
    Readiness:    http-get http://:gerrit-port/config/server/version delay=5s timeout=1s period=1s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/config/gerrit.config from gerrit-config (rw,path="gerrit.config")
      /var/config/replication.config from gerrit-config (rw,path="replication.config")
      /var/config/secure.config from gerrit-secure-config (rw,path="secure.config")
      /var/gerrit from gerrit-site (rw)
      /var/mnt/git from git-filesystem (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pxktr (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  gerrit-site:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  gerrit-site-gerrit-gerrit-stateful-set-0
    ReadOnly:   false
  git-filesystem:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  gerrit-git-filesystem-pvc
    ReadOnly:   false
  gerrit-init-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gerrit-gerrit-init-configmap
    Optional:  false
  gerrit-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gerrit-gerrit-configmap
    Optional:  false
  gerrit-secure-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  gerrit-gerrit-secure-config
    Optional:    false
  default-token-pxktr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-pxktr
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "gerrit-gerrit-stateful-set-0": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "gerrit-gerrit-stateful-set-0": pod has unbound immediate PersistentVolumeClaims

What Could be the missing piece and Does this chart use Default H2 database ?

thomasdraebing commented 4 years ago

Hi @Sjnahak,

the gerrit-gerrit-stateful-set-0-volume uses a RWO-volume. It is used to persist most of the Gerrit site (except for the repositories). Thus it would use the default storageClass, or the one you configured in its stead. The persistent volume claim is looking for a storageClass/Volume with the AccessMode ReadWriteOnce. This is hardcoded in the chart. SInce you want to use a NFS volume, it doesn't find a volume with the spec it is looking for. To make it work, go to the ./helm-charts/gerrit/templates/gerrit.stateful-set.yaml-file and change the AccessMode in line 179 to ReadWriteMany. Then it should work. However, then also other pods might just mount the same volume, which will cause unwanted side effects like concurrent write access. Another solution might be to create a second StorageClass for the NFS provisioner, that declares the AccessMode ReadWriteOnce. That should also work, I think. Since you stated, that you also tried the EBS volume, the best solution would be to use the equivalent in your infrastructure. Which hyperscaler/infrastructure are you using? Cinder for OpenStack or GCEPersistentDisk for GCP would work as well. So do all equivalent block storage types that are supported by Kubernetes [1].

The chart is designed to run Gerrit 3.1 at the moment. This version of Gerrit does not use a relational database anymore. It uses git to store its data (notedb).

I hope that helps, Thomas

PS.: Please consider asking your questions on the Gerrit mailing list: repo-discuss@googlegroups.com. That is the main support platform being used and will reach more people, if they have similar questions. Since GitHub is just a mirror for the repository (development is happening on Gerrit), a lot of people won't check here.

[1] https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes