Use helm install failed

dommgifer commented 4 years ago

Description Follow this doc mariadb-galera to install mariadb-galera on kubernetes.

But mariadb-galera-0 pod always failed.

Check pod log, the message was:

2019-12-31  3:39:05 2 [Note] WSREP: Server status change joined -> synced
2019-12-31  3:39:05 2 [Note] WSREP: Synchronized with group, ready for connections
2019-12-31  3:39:05 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-12-31  3:39:19 16 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:39:29 17 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:39:39 18 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:39:49 19 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:39:59 20 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:40:09 21 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:40:19 22 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:40:29 23 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:40:39 24 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:40:49 25 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:40:55 26 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-12-31  3:40:56 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown

Then, enter into pod, exec command mysql -u root -p$MARIADB_ROOT_PASSWORD, got message'Access denied for user 'root'@'localhost' (using password: YES)' too, and then I tried command mysql, login mariadb success.

Here is my values.yaml

## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry and imagePullSecrets
##
# global:
#   imageRegistry: myRegistryName
#   imagePullSecrets:
#     - myRegistryKeySecretName
#   storageClass: myStorageClass

## Bitnami MariaDB Galera image
## ref: https://hub.docker.com/r/bitnami/mariadb-galera/tags/
##
image:
  registry: docker.io
  repository: bitnami/mariadb-galera
  tag: 10.4.11-debian-9-r9
  ## Specify a imagePullPolicy
  ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistryKeySecretName

  ## Set to true if you would like to see extra information on logs
  ## It turns BASH debugging in minideb-extras-base
  ##
  debug: false

## String to partially override mariadb-galera.fullname template (will maintain the release name)
##
# nameOverride:

## String to fully override mariadb-galera.fullname template
##
# fullnameOverride:

## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:

## Specifies the Kubernetes Cluster's Domain Name.
##
clusterDomain: cluster.local

## MariaDB Gallera K8s svc properties
##
service:
  ## Kubernetes service type and port number
  ##
  type: ClusterIP
  port: 3306
  # clusterIP: None

  ## Specify the nodePort value for the LoadBalancer and NodePort service types.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
  ##
  # nodePort: 30001

  ## Set the LoadBalancer service type to internal only.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
  ##
  # loadBalancerIP:

  ## Load Balancer sources
  ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
  ##
  # loadBalancerSourceRanges:
  # - 10.10.10.0/24

  ## Provide any additional annotations which may be required. This can be used to
  ## set the LoadBalancer service type to internal only.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
  ##
  annotations: {}

## Pods Service Account
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
##
serviceAccount:
  ## Specifies whether a ServiceAccount should be created
  ##
  create: false
  ## The name of the ServiceAccount to use.
  ## If not set and create is true, a name is generated using the mariadb-galera.fullname template
  # name:

## Role Based Access
## Ref: https://kubernetes.io/docs/admin/authorization/rbac/
##
rbac:
  ## Specifies whether RBAC rules should be created
  ##
  create: false

## Pod Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
  enabled: false
  fsGroup: 1001
  runAsUser: 1001

## Database credentials for root (admin) user
##
rootUser:
  ## MariaDB admin password
  ## Password is ignored if existingSecret is specified.
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#setting-the-root-password-on-first-run
  ##
  password: openstack
  ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
  ## If it is not force, a random password will be generated.
  ##
  forcePassword: true

## Use existing secret (ignores rootUser.password, db.password, and galera.mariabackup.password)
##
# existingSecret:

## Custom db configuration
##
db:
  ## MariaDB username and password
  ## Password is ignored if existingSecret is specified.
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#creating-a-database-user-on-first-run
  ##
  user: admin
  password: test
  ## Database to create
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#creating-a-database-on-first-run
  ##
  name: frontend
  ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
  ## If it is not force, a random password will be generated.
  ##
  forcePassword: true

## Galera configuration
##
galera:
  ## Galera cluster name
  ##
  name: galera

  ## Credentials to perform backups
  ##
  mariabackup:
    ## MariaBackup username and password
    ## Password is ignored if existingSecret is specified.
    ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#setting-up-a-multi-master-cluster
    ##
    user: mariabackup
    password: bKCHdpzdSeJAkxq
    ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
    ## If it is not force, a random password will be generated.
    ##
    forcePassword: true

## LDAP configuration
##
ldap:
  ## Enable LDAP support
  ##
  enabled: false
  uri:
  base:
  binddn:
  bindpw:
  bslookup:
  nss_initgroups_ignoreusers: root,nslcd
  scope:
  tls_reqcert:

## Configure MariaDB with a custom my.cnf file
## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file
## Alternatively, you can put your my.cnf under the files/ directory
##
mariadbConfiguration: |-
  [client]
  port=3306
  socket=/opt/bitnami/mariadb/tmp/mysql.sock
  plugin_dir=/opt/bitnami/mariadb/plugin

  [mysqld]
  default-storage-engine=InnoDB
  basedir=/opt/bitnami/mariadb
  datadir=/bitnami/mariadb/data
  plugin_dir=/opt/bitnami/mariadb/plugin
  tmpdir=/opt/bitnami/mariadb/tmp
  socket=/opt/bitnami/mariadb/tmp/mysql.sock
  pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid
  bind-address=0.0.0.0

  ## Character set
  collation-server=utf8_unicode_ci
  init-connect='SET NAMES utf8'
  character-set-server=utf8

  ## MyISAM
  key-buffer-size=32M
  myisam-recover-options=FORCE,BACKUP

  ## safety
  skip-host-cache
  skip-name-resolve
  max-allowed-packet=16M
  max-connect-errors=1000000
  sql-mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY
  sysdate-is-now=1
  innodb=FORCE
  innodb-strict-mode=1
  innodb_file_per_table=1
  # Mandatory per https://github.com/codership/documentation/issues/25
  innodb-autoinc-lock-mode=2
  # Per https://www.percona.com/blog/2006/08/04/innodb-double-write/
  innodb-doublewrite=1
  # Not fully ACID compliant, up to 1sec transaction loss in the event of total cluster failure (across both regions)
  # Enabled for performance per https://mariadb.com/kb/en/mariadb/getting-started-with-mariadb-galera-cluster/
  innodb_flush_log_at_trx_commit=0

  ## binary logging
  log-bin=mysql-bin
  expire-logs-days=14
  # Disabling for performance per http://severalnines.com/blog/9-tips-going-production-galera-cluster-mysql
  sync-binlog=0
  # Required for Galera
  binlog-format=row
  ## Caches and limits
  tmp-table-size=32M
  max-heap-table-size=32M
  # Re-enabling as now works with Maria 10.1.2
  query-cache-type=1
  query-cache-limit=4M
  query-cache-size=256M
  max-connections=500
  thread-cache-size=50
  open-files-limit=65535
  table-definition-cache=4096
  table-open-cache=4096
  ## innodb
  innodb-flush-method=O_DIRECT
  innodb-log-files-in-group=2
  innodb-log-file-size=128M
  innodb-flush-log-at-trx-commit=1
  innodb-file-per-table=1
  # 80% Memory is default reco.
  # Need to re-evaluate when DB size grows
  innodb-buffer-pool-size=2G
  innodb_file_format=Barracuda

  ## logging
  log-error=/opt/bitnami/mariadb/logs/mysqld.log
  slow-query-log-file=/opt/bitnami/mariadb/logs/mysqld.log
  log-queries-not-using-indexes=1
  slow-query-log=1

  [galera]
  wsrep_on=ON
  wsrep_provider=/opt/bitnami/mariadb/lib/libgalera_smm.so
  wsrep_sst_method=mariabackup
  wsrep_slave_threads=4
  wsrep_cluster_address=gcomm://
  wsrep_cluster_name=galera
  wsrep_sst_auth="root:"
  innodb-flush-log-at-trx-commit=2
  # MYISAM REPLICATION SUPPORT #
  wsrep_replicate_myisam=ON

  [mariadb]
  plugin_load_add=auth_pam

## ConfigMap with MariaDB configuration
## NOTE: This will override mariadbConfiguration
##
# configurationConfigMap:

## initdb scripts
## Specify dictionary of scripts to be run at first boot
## Alternatively, you can put your scripts under the files/docker-entrypoint-initdb.d directory
##
# initdbScripts:
#   my_init_script.sh: |
#      #!/bin/sh
#      echo "Do something."

## ConfigMap with scripts to be run at first boot
## Note: This will override initdbScripts
##
# initdbScriptsConfigMap:

## MariaDB additional command line flags
## Can be used to specify command line flags, for example:
##
## extraFlags: "--max-connect-errors=1000 --max_connections=155"

## Desired number of cluster nodes
##
replicaCount: 3

## updateStrategy for MariaDB Master StatefulSet
## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
##
updateStrategy:
  type: RollingUpdate

## Additional pod annotations for MariaDB Galera pods
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
##
podAnnotations: {}

## Affinity for pod assignment. Evaluated as a template.
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}

## Node labels for pod assignment. Evaluated as a template.
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}

## Tolerations for pod assignment. Evaluated as a template.
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: {}

## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  ## If true, use a Persistent Volume Claim, If false, use emptyDir
  ##
  enabled: true
  # Enable persistence using an existing PVC
  # existingClaim:
  # Subdirectory of the volume to mount
  # subPath:
  mountPath: /bitnami/mariadb
  ## Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # storageClass: "-"
  ## Persistent Volume Claim annotations
  ##
  annotations:
  ## Persistent Volume Access Mode
  ##
  accessModes:
    - ReadWriteOnce
  ## Persistent Volume size
  ##
  size: 8Gi

## Additional init containers
##
extraInitContainers: []
# - name: do-something
#   image: bitnami/minideb:stretch
#   command: ['do', 'something']

## Additional containers
##
extraContainers: []

## MariaDB Galera containers' resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  limits: {}
  #   cpu: 0.5
  #   memory: 256Mi
  requests: {}
  #   cpu: 0.5
  #   memory: 256Mi

## MariaDB Galera containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
##
livenessProbe:
  enabled: true
  ## Initializing the database could take some time
  ##
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
readinessProbe:
  enabled: true
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3

## Pod disruption budget configuration
##
podDisruptionBudget:
  ## Specifies whether a Pod disruption budget should be created
  ##
  create: false
  minAvailable: 1
  # maxUnavailable: 1

## Prometheus exporter configuration
##
metrics:
  enabled: false
  ## Bitnami MySQL Prometheus exporter image
  ## ref: https://hub.docker.com/r/bitnami/mysqld-exporter/tags/
  ##
  image:
    registry: docker.io
    repository: bitnami/mysqld-exporter
    tag: 0.12.1-debian-9-r136
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistryKeySecretName
  ## MySQL Prometheus exporter containers' resource requests and limits
  ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
  ##
  resources:
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    limits: {}
    #   cpu: 0.5
    #   memory: 256Mi
    requests: {}
    #   cpu: 0.5
    #   memory: 256Mi
  ## MySQL Prometheus exporter service parameters
  ##
  service:
    type: ClusterIP
    port: 9104
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9104"

  ## Prometheus Operator ServiceMonitor configuration
  ##
  serviceMonitor:
    enabled: false
    ## Namespace in which Prometheus is running
    ##
    # namespace: monitoring

    ## Interval at which metrics should be scraped.
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
    ##
    # interval: 10s

    ## Timeout after which the scrape is ended
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
    ##
    # scrapeTimeout: 10s

    ## ServiceMonitor selector labels
    ## ref: https://github.com/bitnami/charts/tree/master/bitnami/prometheus-operator#prometheus-configuration
    ##
    selector:
      prometheus: kube-prometheus

Steps to reproduce the issue:

helm install --name mariadb-galera -f values.yaml bitnami/mariadb-galera

Describe the results you received: Here is pod log

Subscribe to project updates by watching  
Submit issues and feature requests at  
Send us your feedback at  

 ==> ** Starting MariaDB setup ** 
 ==> Validating settings in MYSQL_*/MARIADB_* env vars.. 
 ==> Initializing mariadb database... 
 ==> Persisted data detected. Restoring... 
 ==> ** MariaDB setup finished! ** 

 ==> ** Starting MariaDB ** 
2019-12-31  3:38:51 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.4.11-MariaDB-log) starting as process 1 ... 
/opt/bitnami/mariadb/sbin/mysqld: Can't create file '/opt/bitnami/mariadb/logs/mysqld.log' (errno: 13 "Permission denied") 
2019-12-31  3:38:51 0 [Note] WSREP: Loading provider /opt/bitnami/mariadb/lib/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1 
wsrep loader: [INFO] wsrep_load(): loading provider library '/opt/bitnami/mariadb/lib/libgalera_smm.so' 
wsrep loader: [INFO] wsrep_load(): Galera 4.3(r0) by Codership Oy <info@codership.com> loaded successfully. 
2019-12-31  3:38:51 0 [Note] WSREP: CRC-32C: using hardware acceleration. 
2019-12-31  3:38:51 0 [Note] WSREP: Found saved state: f2f454ec-2b7e-11ea-8cc8-3bdd0c967da2:1, safe_to_bootstrap: 1 
2019-12-31  3:38:51 0 [Note] WSREP: GCache DEBUG: opened preamble: 
Version: 2 
UUID: 00000000-0000-0000-0000-000000000000 
Seqno: 1 - 1 
Offset: 1280 
Synced: 1 
2019-12-31  3:38:51 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID. 
2019-12-31  3:38:51 0 [Note] WSREP: Passing config to GCS: base_dir = /bitnami/mariadb/data/; base_host = 10.42.4.35; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /bitnami/mariadb/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc 
2019-12-31  3:38:52 0 [Note] WSREP: Service thread queue flushed. 
2019-12-31  3:38:52 0 [Note] WSREP: ####### Assign initial position for certification: f2f454ec-2b7e-11ea-8cc8-3bdd0c967da2:1, protocol version: -1 
2019-12-31  3:38:52 0 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0 -> f2f454ec-2b7e-11ea-8cc8-3bdd0c967da2:1 
2019-12-31  3:38:52 0 [Note] WSREP: Start replication 
2019-12-31  3:38:52 0 [Note] WSREP: Connecting with bootstrap option: 1 
2019-12-31  3:38:52 0 [Note] WSREP: Setting GCS initial position to f2f454ec-2b7e-11ea-8cc8-3bdd0c967da2:1 
2019-12-31  3:38:52 0 [Note] WSREP: protonet asio version 0 
2019-12-31  3:38:52 0 [Note] WSREP: Using CRC-32C for message checksums. 
2019-12-31  3:38:52 0 [Note] WSREP: backend: asio 
2019-12-31  3:38:52 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 
2019-12-31  3:38:52 0 [Warning] WSREP: access file(/bitnami/mariadb/data//gvwstate.dat) failed(No such file or directory) 
2019-12-31  3:38:52 0 [Note] WSREP: restore pc from disk failed 
2019-12-31  3:38:52 0 [Note] WSREP: GMCast version 0 
2019-12-31  3:38:52 0 [Note] WSREP: (104f8689, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 
2019-12-31  3:38:52 0 [Note] WSREP: (104f8689, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 
2019-12-31  3:38:52 0 [Note] WSREP: EVS version 1 
2019-12-31  3:38:52 0 [Note] WSREP: gcomm: bootstrapping new group 'galera' 
2019-12-31  3:38:52 0 [Note] WSREP: start_prim is enabled, turn off pc_recovery 
2019-12-31  3:38:52 0 [Note] WSREP: EVS version upgrade 0 -> 1 
2019-12-31  3:38:52 0 [Note] WSREP: PC protocol upgrade 0 -> 1 
2019-12-31  3:38:52 0 [Note] WSREP: Node 104f8689 state prim 
2019-12-31  3:38:52 0 [Note] WSREP: view(view_id(PRIM,104f8689,1) memb { 
    104f8689,0 
} joined { 
} left { 
} partitioned { 
}) 
2019-12-31  3:38:52 0 [Note] WSREP: save pc into disk 
2019-12-31  3:38:52 0 [Note] WSREP: gcomm: connected 
2019-12-31  3:38:52 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 
2019-12-31  3:38:52 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 
2019-12-31  3:38:52 0 [Note] WSREP: Opened channel 'galera' 
2019-12-31  3:38:52 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1 
2019-12-31  3:38:52 1 [Note] WSREP: Starting rollbacker thread 1 
2019-12-31  3:38:52 2 [Note] WSREP: Starting applier thread 2 
2019-12-31  3:38:52 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 10926247-2b7f-11ea-8edd-5be84ff54863 
2019-12-31  3:38:52 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 10926247-2b7f-11ea-8edd-5be84ff54863 
2019-12-31  3:38:52 0 [Note] WSREP: STATE EXCHANGE: got state msg: 10926247-2b7f-11ea-8edd-5be84ff54863 from 0 (galera-mariadb-galera-0) 
2019-12-31  3:38:52 0 [Note] WSREP: Quorum results: 
    version    = 6, 
    component  = PRIMARY, 
    conf_id    = 0, 
    members    = 1/1 (joined/total), 
    act_id     = 1, 
    last_appl. = 1, 
    protocols  = 2/10/4 (gcs/repl/appl), 
    vote policy= 0, 
    group UUID = f2f454ec-2b7e-11ea-8cc8-3bdd0c967da2 
2019-12-31  3:38:52 0 [Note] WSREP: Flow-control interval: [16, 16] 
2019-12-31  3:38:52 0 [Note] WSREP: Restored state OPEN -> JOINED (2) 
2019-12-31  3:38:52 0 [Note] WSREP: Member 0.0 (galera-mariadb-galera-0) synced with group. 
2019-12-31  3:38:52 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 2) 
2019-12-31  3:38:52 2 [Note] WSREP: ####### processing CC 2, local, ordered 
2019-12-31  3:38:52 2 [Note] WSREP: REPL Protocols: 10 (5, 3) 
2019-12-31  3:38:52 2 [Note] WSREP: ####### My UUID: 104f8689-2b7f-11ea-8586-c6445b48a3b4 
2019-12-31  3:38:52 2 [Note] WSREP: Server galera-mariadb-galera-0 connected to cluster at position f2f454ec-2b7e-11ea-8cc8-3bdd0c967da2:2 with ID 104f8689-2b7f-11ea-8586-c6445b48a3b4 
2019-12-31  3:38:52 2 [Note] WSREP: Server status change disconnected -> connected 
2019-12-31  3:38:52 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:38:52 2 [Note] WSREP: ####### ST not required 
2019-12-31  3:38:52 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no 
2019-12-31  3:38:52 0 [Note] WSREP: Service thread queue flushed. 
2019-12-31  3:38:52 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5 
2019-12-31  3:38:52 2 [Note] WSREP: ####### Adjusting cert position: -1 -> 2 
2019-12-31  3:38:52 0 [Note] WSREP: Service thread queue flushed. 
2019-12-31  3:38:52 2 [Note] WSREP: ####### Setting monitor position to 2 
2019-12-31  3:38:53 2 [Note] WSREP: Lowest cert indnex boundary for CC from group: 2 
2019-12-31  3:38:53 2 [Note] WSREP: Min available from gcache for CC from group: 2 
2019-12-31  3:38:53 2 [Note] WSREP: ================================================ 
View: 
  id: f2f454ec-2b7e-11ea-8cc8-3bdd0c967da2:2 
  status: primary 
  protocol_version: 4 
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO 
  final: no 
  own_index: 0 
  members(1): 
    0: 104f8689-2b7f-11ea-8586-c6445b48a3b4, galera-mariadb-galera-0 
================================================= 
2019-12-31  3:38:53 2 [Note] WSREP: Server status change connected -> joiner 
2019-12-31  3:38:53 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:38:53 2 [Note] WSREP: Server status change joiner -> initializing 
2019-12-31  3:38:53 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:38:53 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/ 
2019-12-31  3:38:53 0 [Note] InnoDB: Using Linux native AIO 
2019-12-31  3:38:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 
2019-12-31  3:38:53 0 [Note] InnoDB: Uses event mutexes 
2019-12-31  3:38:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.8 
2019-12-31  3:38:53 0 [Note] InnoDB: Number of pools: 1 
2019-12-31  3:38:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 
2019-12-31  3:38:53 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts) 
2019-12-31  3:38:53 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M 
2019-12-31  3:38:53 0 [Note] InnoDB: Completed initialization of buffer pool 
2019-12-31  3:38:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 
2019-12-31  3:38:54 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=75546 
2019-12-31  3:38:59 0 [Note] InnoDB: Starting final batch to recover 16 pages from redo log. 
2019-12-31  3:39:00 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 
2019-12-31  3:39:00 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 
2019-12-31  3:39:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables 
2019-12-31  3:39:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 
2019-12-31  3:39:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 
2019-12-31  3:39:00 0 [Note] InnoDB: Waiting for purge to start 
2019-12-31  3:39:00 0 [Note] InnoDB: 10.4.11 started; log sequence number 75735; transaction id 46 
2019-12-31  3:39:00 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool 
2019-12-31  3:39:00 0 [Note] Plugin 'FEEDBACK' is disabled. 
2019-12-31  3:39:00 0 [Note] InnoDB: Buffer pool(s) load completed at 191231  3:39:00 
2019-12-31  3:39:01 0 [Note] Server socket created on IP: '0.0.0.0'. 
2019-12-31  3:39:01 0 [Warning] 'user' entry 'root@galera-mariadb-galera-0' ignored in --skip-name-resolve mode. 
2019-12-31  3:39:01 0 [Warning] 'user' entry '@galera-mariadb-galera-0' ignored in --skip-name-resolve mode. 
2019-12-31  3:39:01 0 [Warning] 'proxies_priv' entry '@% root@galera-mariadb-galera-0' ignored in --skip-name-resolve mode. 
2019-12-31  3:39:01 0 [Note] WSREP: wsrep_init_schema_and_SR (nil) 
2019-12-31  3:39:01 0 [Note] WSREP: Server initialized 
2019-12-31  3:39:01 0 [Note] WSREP: Server status change initializing -> initialized 
2019-12-31  3:39:01 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:39:01 2 [Note] WSREP: Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1 
2019-12-31  3:39:02 10 [Note] WSREP: Recovered cluster id f2f454ec-2b7e-11ea-8cc8-3bdd0c967da2 
2019-12-31  3:39:02 2 [Note] WSREP: Server status change initialized -> joined 
2019-12-31  3:39:02 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:39:02 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:39:02 0 [Note] Reading of all Master_info entries succeeded 
2019-12-31  3:39:02 0 [Note] Added new Master_info '' to hash table 
2019-12-31  3:39:02 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections. 
Version: '10.4.11-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution 
2019-12-31  3:39:02 13 [Note] WSREP: Starting applier thread 13 
2019-12-31  3:39:02 14 [Note] WSREP: Starting applier thread 14 
2019-12-31  3:39:02 15 [Note] WSREP: Starting applier thread 15 
2019-12-31  3:39:05 2 [Note] WSREP: Server galera-mariadb-galera-0 synced with group 
2019-12-31  3:39:05 2 [Note] WSREP: Server status change joined -> synced 
2019-12-31  3:39:05 2 [Note] WSREP: Synchronized with group, ready for connections 
2019-12-31  3:39:05 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:39:19 16 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:39:29 17 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:39:39 18 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:39:49 19 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:39:59 20 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:40:09 21 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:40:19 22 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:40:29 23 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:40:39 24 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:40:49 25 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:40:55 26 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 
2019-12-31  3:40:56 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown 
2019-12-31  3:40:56 0 [Note] WSREP: Shutdown replication 
2019-12-31  3:40:56 0 [Note] WSREP: Server status change synced -> disconnecting 
2019-12-31  3:40:56 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:40:56 0 [Note] WSREP: Closing send monitor... 
2019-12-31  3:40:56 0 [Note] WSREP: Closed send monitor. 
2019-12-31  3:40:56 0 [Note] WSREP: gcomm: terminating thread 
2019-12-31  3:40:56 0 [Note] WSREP: gcomm: joining thread 
2019-12-31  3:40:56 0 [Note] WSREP: gcomm: closing backend 
2019-12-31  3:40:56 0 [Note] WSREP: PC protocol downgrade 1 -> 0 
2019-12-31  3:40:56 0 [Note] WSREP: view((empty)) 
2019-12-31  3:40:56 0 [Note] WSREP: gcomm: closed 
2019-12-31  3:40:56 0 [Note] WSREP: New SELF-LEAVE. 
2019-12-31  3:40:56 0 [Note] WSREP: Flow-control interval: [0, 0] 
2019-12-31  3:40:56 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 
2019-12-31  3:40:56 0 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 2) 
2019-12-31  3:40:56 0 [Note] WSREP: RECV thread exiting 0: Success 
2019-12-31  3:40:56 2 [Note] WSREP: ####### processing CC -1, local, ordered 
2019-12-31  3:40:56 2 [Note] WSREP: ####### My UUID: 104f8689-2b7f-11ea-8586-c6445b48a3b4 
2019-12-31  3:40:56 2 [Note] WSREP: ####### ST not required 
2019-12-31  3:40:56 0 [Note] WSREP: recv_thread() joined. 
2019-12-31  3:40:56 0 [Note] WSREP: Closing replication queue. 
2019-12-31  3:40:56 0 [Note] WSREP: Closing slave action queue. 
2019-12-31  3:40:56 2 [Note] WSREP: ================================================ 
View: 
  id: f2f454ec-2b7e-11ea-8cc8-3bdd0c967da2:-1 
  status: non-primary 
  protocol_version: 4 
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO 
  final: yes 
  own_index: -1 
  members(0): 
================================================= 
2019-12-31  3:40:56 2 [Note] WSREP: Non-primary view 
2019-12-31  3:40:56 2 [Note] WSREP: Server status change disconnecting -> disconnected 
2019-12-31  3:40:56 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:40:56 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 
2019-12-31  3:40:56 15 [Note] WSREP: Applier thread exiting ret: 6 thd: 15 
2019-12-31  3:40:56 15 [Warning] Aborted connection 15 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) 
2019-12-31  3:40:56 13 [Note] WSREP: Applier thread exiting ret: 6 thd: 13 
2019-12-31  3:40:56 13 [Warning] Aborted connection 13 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) 
2019-12-31  3:40:56 14 [Note] WSREP: Applier thread exiting ret: 6 thd: 14 
2019-12-31  3:40:56 14 [Warning] Aborted connection 14 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) 
2019-12-31  3:40:57 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2 
2019-12-31  3:40:57 1 [Note] WSREP: rollbacker thread exiting 1 
2019-12-31  3:40:57 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) 
2019-12-31  3:40:57 1 [Warning] Aborted connection 1 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) 
2019-12-31  3:40:57 0 [Note] Event Scheduler: Purging the queue. 0 events 
2019-12-31  3:40:57 0 [Note] InnoDB: FTS optimize thread exiting. 
2019-12-31  3:40:57 0 [Note] WSREP: dtor state: CLOSED 
2019-12-31  3:40:57 0 [Note] WSREP: MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0 
2019-12-31  3:40:57 0 [Note] WSREP: apply mon: entered 0 
2019-12-31  3:40:57 0 [Note] WSREP: apply mon: entered 0 
2019-12-31  3:40:57 0 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0 
2019-12-31  3:40:57 0 [Note] WSREP: cert index usage at exit 0 
2019-12-31  3:40:57 0 [Note] WSREP: cert trx map usage at exit 0 
2019-12-31  3:40:57 0 [Note] WSREP: deps set usage at exit 0 
2019-12-31  3:40:57 0 [Note] WSREP: avg deps dist 0 
2019-12-31  3:40:57 0 [Note] WSREP: avg cert interval 0 
2019-12-31  3:40:57 0 [Note] WSREP: cert index size 0 
2019-12-31  3:40:57 0 [Note] WSREP: Service thread queue flushed. 
2019-12-31  3:40:57 0 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0 
2019-12-31  3:40:57 0 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0 
2019-12-31  3:40:57 0 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 2) 
2019-12-31  3:40:57 0 [Note] WSREP: Flushing memory map to disk... 
2019-12-31  3:40:58 0 [Note] InnoDB: Starting shutdown... 
2019-12-31  3:40:58 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool 
2019-12-31  3:40:58 0 [Note] InnoDB: Buffer pool(s) dump completed at 191231  3:40:58 
2019-12-31  3:41:02 0 [Note] InnoDB: Shutdown completed; log sequence number 82559; transaction id 61 
2019-12-31  3:41:02 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 
2019-12-31  3:41:03 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete

Describe the results you expected: I expected the three mariadb-galera pod running.

Version

Output of docker version:

18.09.9

Output of docker info:

Server:
 Containers: 27
  Running: 14
  Paused: 0
  Stopped: 13
 Images: 10
 Server Version: 18.09.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-64-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.852GiB
 Name: master-1
 ID: BCP3:FRZZ:HO6R:M3EH:LEYJ:DOL6:V2K7:B4R6:6ECD:FJBU:HGHB:57KY
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional environment details (AWS, VirtualBox, Docker for MAC, physical, etc.): Kubernetes version: v1.16.3

carrodher commented 4 years ago

Hi, I am sorry but I am not able to reproduce your issue. This is what I have done in order to try to reproduce your scenario:

Copy your values.yaml in a file with the same name in my current directory.
Then install the chart using the file: helm install mariadb-galera -f values.yaml bitnami/mariadb-galera

Once everything deployed, check the logs of the mariadb-galera-0 and I didn't found anything weird. After that, I accessed this pod and checked the connection with the database:


▶ kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
mariadb-galera-0   1/1     Running   0          3m46s
mariadb-galera-1   1/1     Running   0          2m37s
mariadb-galera-2   1/1     Running   0          95s

▶ kubectl exec -it mariadb-galera-0 /bin/bash root@mariadb-galera-0:/# mysql -u root -p$MARIADB_ROOT_PASSWORD Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 46 Server version: 10.4.11-MariaDB-log Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>


I did the installation from scratch, please take into account that `helm delete` doesn't remove the PVCs, so maybe if you did a previous installation in the past using another password (or the randomly generated one), the old PVC is mounted into the new deployment and there is any inconsistency between the new and old password. If that is the case, please try again removing the PVC. Example of this:

▶ helm delete mariadb-galera release "mariadb-galera" uninstalled

▶ kubectl get pods No resources found.

▶ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-mariadb-galera-0 Bound pvc-4343205f-2b9f-11ea-b733-42010a9600a6 8Gi RWO standard 10m data-mariadb-galera-1 Bound pvc-6c79fd69-2b9f-11ea-b733-42010a9600a6 8Gi RWO standard 9m22s data-mariadb-galera-2 Bound pvc-91b3502f-2b9f-11ea-b733-42010a9600a6 8Gi RWO standard 8m19s

davidjeddy commented 4 years ago

@dommgifer I also ran into the same issue. So I ran helm install ... in a couple different permutations. My values.yaml is directly from https://github.com/bitnami/charts/blob/master/bitnami/mariadb-galera/values.yaml with no changes.

command | result helm install --name mariadb-galera-fv -f values.yaml bitnami/mariadb-galera --version 0.6.1 | worked helm install --name mariadb-galera-f -f values.yaml bitnami/mariadb-galera | worked helm install --name mariadb-galera-v bitnami/mariadb-galera --version 0.6.1 | worked helm install --name mariadb-galera bitnami/mariadb-galera | worked

BUT when I install via this command:

export RELEASE_NAME="mariadb-galera-cluster"
helm install --name ${RELEASE_NAME} -f values.yaml bitnami/mariadb-galera --version 0.6.1

...the install works as expected.

However, if I set RELEASE_NAME to mariadb-galera, the install fails.

So I re-ran

helm install --name mariadb-galera-fv -f values.yaml bitnami/mariadb-galera --version 0.6.1

... and it failed as well. I got a wild idea: what about the PVC's? So I deleted all of them...

kubectl get pvc |  awk '$1 {print$1}' | while read vol; do kubectl delete pvc/${vol}; done

AND it worked. So I deleted the helm depolyment and re-launched it. Failed.

Finds

Not deleting the PVC between helm installs will cause the pod to fail; thus causing the the service to never become ready.

Versions

docker --version: Docker version 19.03.5, build 633a0ea838
kubectl version: Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
helm version: Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
minikube version:minikube version: v1.4.0 commit: 7969c25a98a018b94ea87d949350f3271e9d64b6

dommgifer commented 4 years ago

I always delete pvc before reinstall mariadb-galera. But still not worked.

So, I try to install mariadb-galera with :

persistence:
  enable: false

And it worked.

@carrodher @davidjeddy What kind of back-end storage on your environment?

My back-end storage was NFS.

carrodher commented 4 years ago

It is weird, I mean, the source of the issue is that the old password is stored anywhere and the new deployment is trying to use the old password instead of the new one, but this issue should disappear if the PVCs are deleted.

In my case, I have the cluster in Google using the disk provided by them, so it is not NFS for sure.

ekhaydarov commented 4 years ago

the issue is that the new password is not mounted for postgres because postgres has the PVC which already has a config directory. therefore bitnami doesnt look to reload the config on helm upgrade if persistence is enabled.

to solve this issue you either persist the secret or you clear the postgres config directory so a new secret can be mounted. you could at the very beginning set your postgresqlPassword so it never changes across upgrades. as well as disabling persistence

deleting the PVCs every time is impractical, as its a feature of helm. it is also against helm upgrade requirements as all youre doing is updating the charts.

Edit: its actually a known thing and is in the comments of the values.yaml it would be great to make the user aware of these issues in the README

davidjeddy commented 4 years ago

@dommgifer I am using minikube with k8s 1.13 on an Ubuntu 18.04 host. The storage class is the standard storage.

@ekhaydarov Excellent insight, thank you. That helps to explain the situation very nicely.

stale[bot] commented 4 years ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

stale[bot] commented 4 years ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

stevefan1999-personal commented 4 years ago

/reopen

I have found a way to workarouind this. Basically you do a brute force attack -- setting the password yourself. You will need to connect to the TTY of the main MariaDB pod, and type the following commands:

$ mysql -u root <enter>

If done right you will be greeted with MariaDB welcome message. You can hop around, use SHOW database and stuff. This indicates your password actually aren't being set by the initial launch script on Bitnami side. It's clearly an oversight to be fixed, I'll see what to do later. Then, as we have obtained the root privilege, we can add alter the privilege information as we do in textbook, execute the following SQL commands:

ALTER USER 'root'@'localhost' IDENTIFIED BY '${rootUser.password}';
ALTER USER 'root'@'127.0.0.1' IDENTIFIED BY '${rootUser.password}';

Replace ${rootUser.password} to the actual password you set/generated from the Helm chart. Everything should go back to normal since then. It's so stupid that Bitnami couldn't even do a proper check whenever they push a change albeit this kind of low level error could still present. I see that they commit so frequently and so does their Galera charts.

stevefan1999-personal commented 4 years ago

But coming to think about this, I think MariaDB prefer us NOT to use native socket with local administration, and actually prefers using UDS instead. So I think this is why I can connect to the server successfully without any password set. See https://superuser.com/a/1103735 for more. SUGGESTION 1: Disable/Do not use UDS auth plugin from the upstream Bitnami MariaDB Docker image. This could be done here. SUGGESTION 2: Add the two lines I suggested on script of the permalink also mentioned above.

carrodher commented 4 years ago

At the end the issue is that with the default options, the random password is stored in the PV then when you uninstall the deployment with helm delete the PV/PVCs are not deleted. So the second installation generates a new random password different from the first one that is still present in the PV so there is a conflict between the new password and the previous one.

This is not a specific issue related to the MariaDB chart; we are thinking of a solution/clarification for this topic.

mn0o7 commented 4 years ago

Hi,

Im having the same issue it looks like somehow installation of mariadb in a different namespace is messing with me. Current installation in namespace: stg name: stg-mariadb-galera

New installation namespace: dev name: dev-mariadb-galera

for both im using storageClass: "rook-ceph-block"

The New installation failed when i used the name dev-mariadb-galera, if i used a different name like dev-xxx it works, if i turn off persistancy it works, if i change root force password to false it works.? My guess would be its looking for something in the pv or somewhere that is related to the name mariadb-galera.

Note that i didnt have any of these issues with my first installation in namespace: stg

I really like keeping my naming convention any help would be much appreciated

mn0o7 commented 4 years ago

Looks like adding this from here extraEnvVars:

name: MARIADB_INIT_SLEEP_TIME value: "20"

andresbono commented 4 years ago

Hi @mn0o7, so adding that variable solved the issue? Isn't related to the PVs, right?

mn0o7 commented 4 years ago

Hi @andresbono My issue is solved but i cant really say what this is related to =

carrodher commented 4 years ago

I can't see the relation between the MARIADB_INIT_SLEEP_TIME env. variable and the issue, so maybe what makes the difference was the restart or reapply the values. Anycase, we're glad that you were able to fix the issue

mn0o7 commented 4 years ago

mariadb 15:28:16.23
mariadb 15:28:16.23 Welcome to the Bitnami mariadb-galera container
mariadb 15:28:16.23 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
mariadb 15:28:16.24 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
mariadb 15:28:16.24
mariadb 15:28:16.24 INFO  > ** Starting MariaDB setup **
mariadb 15:28:16.26 INFO  > Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 15:28:16.34 INFO  > Initializing mariadb database
mariadb 15:28:16.36 WARN  > The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable or does not exist. Configurations based on environment variables will not be applied for this file.
mariadb 15:28:16.36 INFO  > Persisted data detected. Restoring
mariadb 15:28:16.40 INFO  > ** MariaDB setup finished! **

mariadb 15:28:16.46 INFO  > ** Starting MariaDB **
mariadb 15:28:16.46 INFO  > Setting previous boot
2020-08-04 15:28:16 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.4.13-MariaDB-log) starting as process 1 ...
2020-08-04 15:28:16 0 [Note] WSREP: Loading provider /opt/bitnami/mariadb/lib/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
wsrep loader: [INFO] wsrep_load(): loading provider library '/opt/bitnami/mariadb/lib/libgalera_smm.so'
wsrep loader: [INFO] wsrep_load(): Galera 4.5(r0) by Codership Oy <info@codership.com> loaded successfully.
2020-08-04 15:28:16 0 [Note] WSREP: CRC-32C: using hardware acceleration.
2020-08-04 15:28:16 0 [Note] WSREP: Found saved state: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731, safe_to_bootstrap: 0
2020-08-04 15:28:16 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 2
UUID: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b
Seqno: 1402 - 1731
Offset: 1280
Synced: 1
2020-08-04 15:28:16 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b, offset: 1280
2020-08-04 15:28:16 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
2020-08-04 15:28:16 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2020-08-04 15:28:16 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 1402-1731
2020-08-04 15:28:16 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...  0.0% (        0/134202624 bytes) complete.
2020-08-04 15:28:16 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (134202624/134202624 bytes) complete.
2020-08-04 15:28:16 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 37/367 locked buffers
2020-08-04 15:28:16 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): free space: 23912/134217728
2020-08-04 15:28:16 0 [Note] WSREP: Passing config to GCS: base_dir  /bitnami/mariadb/data/; base_host  100.124.161.47; base_port  4567; cert.log_conflicts  no; cert.optimistic_pa  yes; debug  no; evs.auto_evict  0; evs.delay_margin  PT1S; evs.delayed_keep_period  PT30S; evs.inactive_check_period  PT0.5S; evs.inactive_timeout  PT15S; evs.join_retrans_period  PT1S; evs.max_install_timeouts  3; evs.send_window  4; evs.stats_report_period  PT1M; evs.suspect_timeout  PT5S; evs.user_send_window  2; evs.view_forget_timeout  PT24H; gcache.dir  /bitnami/mariadb/data/; gcache.keep_pages_size  0; gcache.mem_size  0; gcache.name  galera.cache; gcache.page_size  128M; gcache.recover  yes; gcache.size  128M; gcomm.thread_prio  ; gcs.fc_debug  0; gcs.fc_factor  1.0; gcs.fc_limit  16; gcs.fc_master_slave  no; gcs.max_packet_size  64500; gcs.max_throttle  0.25; gcs.recv_q_hard_limit  9223372036854775807; gcs.recv_q_soft_limit  0.25; gcs.sync_donor  no; gmcast.segment  0; gmcast.version  0; pc.announce_timeout  P...
2020-08-04 15:28:16 0 [Note] WSREP: Service thread queue flushed.
2020-08-04 15:28:16 0 [Note] WSREP: ####### Assign initial position for certification: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731, protocol version: -1
2020-08-04 15:28:16 0 [Note] WSREP: Start replication
2020-08-04 15:28:16 0 [Note] WSREP: Connecting with bootstrap option: 0
2020-08-04 15:28:16 0 [Note] WSREP: Setting GCS initial position to 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731
2020-08-04 15:28:16 0 [Note] WSREP: protonet asio version 0
2020-08-04 15:28:16 0 [Note] WSREP: Using CRC-32C for message checksums.
2020-08-04 15:28:16 0 [Note] WSREP: backend: asio
2020-08-04 15:28:16 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2020-08-04 15:28:16 0 [Warning] WSREP: access file(/bitnami/mariadb/data//gvwstate.dat) failed(No such file or directory)
2020-08-04 15:28:16 0 [Note] WSREP: restore pc from disk failed
2020-08-04 15:28:16 0 [Note] WSREP: GMCast version 0
2020-08-04 15:28:16 0 [Note] WSREP: (1e4b81b8-bbba, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2020-08-04 15:28:16 0 [Note] WSREP: (1e4b81b8-bbba, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2020-08-04 15:28:16 0 [Note] WSREP: EVS version 1
2020-08-04 15:28:16 0 [Note] WSREP: gcomm: connecting to group 'galera', peer 'stg-mariadb-galera-headless.stg.svc.cluster.local:'
2020-08-04 15:28:16 0 [Note] WSREP: (1e4b81b8-bbba, 'tcp://0.0.0.0:4567') connection established to ab31edb0-ab39 tcp://100.115.157.230:4567
2020-08-04 15:28:16 0 [Note] WSREP: (1e4b81b8-bbba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2020-08-04 15:28:17 0 [Note] WSREP: EVS version upgrade 0 -> 1
2020-08-04 15:28:17 0 [Note] WSREP: declaring ab31edb0-ab39 at tcp://100.115.157.230:4567 stable
2020-08-04 15:28:17 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2020-08-04 15:28:17 0 [Note] WSREP: Node ab31edb0-ab39 state prim
2020-08-04 15:28:17 0 [Note] WSREP: view(view_id(PRIM,1e4b81b8-bbba,4) memb {
        1e4b81b8-bbba,0
        ab31edb0-ab39,0
} joined {
} left {
} partitioned {
})
2020-08-04 15:28:17 0 [Note] WSREP: save pc into disk
2020-08-04 15:28:17 0 [Note] WSREP: gcomm: connected
2020-08-04 15:28:17 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2020-08-04 15:28:17 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2020-08-04 15:28:17 0 [Note] WSREP: Opened channel 'galera'
2020-08-04 15:28:17 0 [Note] WSREP: New COMPONENT: primary  yes, bootstrap  no, my_idx  0, memb_num  2
2020-08-04 15:28:17 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 1ee7e32f-d667-11ea-9e2f-36ab219db152
2020-08-04 15:28:17 1 [Note] WSREP: Starting rollbacker thread 1
2020-08-04 15:28:17 2 [Note] WSREP: Starting applier thread 2
2020-08-04 15:28:17 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 1ee7e32f-d667-11ea-9e2f-36ab219db152
2020-08-04 15:28:17 0 [Note] WSREP: STATE EXCHANGE: got state msg: 1ee7e32f-d667-11ea-9e2f-36ab219db152 from 0 (stg-mariadb-galera-1)
2020-08-04 15:28:17 0 [Note] WSREP: STATE EXCHANGE: got state msg: 1ee7e32f-d667-11ea-9e2f-36ab219db152 from 1 (stg-mariadb-galera-0)
2020-08-04 15:28:17 0 [Note] WSREP: Quorum results:
        version     6,
        component   PRIMARY,
        conf_id     3,
        members     1/2 (joined/total),
        act_id      1732,
        last_appl.  1729,
        protocols   2/10/4 (gcs/repl/appl),
        vote policy 0,
        group UUID  6cf9ed1b-d54b-11ea-8cf7-5fde2825032b
2020-08-04 15:28:17 0 [Note] WSREP: Flow-control interval: [23, 23]
2020-08-04 15:28:17 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1733)
2020-08-04 15:28:17 2 [Note] WSREP: ####### processing CC 1733, local, ordered
2020-08-04 15:28:17 2 [Note] WSREP: Process first view: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b my uuid: 1e4b81b8-d667-11ea-bbba-a6360c66a8a4
2020-08-04 15:28:17 2 [Note] WSREP: Server stg-mariadb-galera-1 connected to cluster at position 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1733 with ID 1e4b81b8-d667-11ea-bbba-a6360c66a8a4
2020-08-04 15:28:17 2 [Note] WSREP: Server status change disconnected -> connected
2020-08-04 15:28:17 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-08-04 15:28:17 2 [Note] WSREP: ####### My UUID: 1e4b81b8-d667-11ea-bbba-a6360c66a8a4
2020-08-04 15:28:17 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2020-08-04 15:28:17 0 [Note] WSREP: Service thread queue flushed.
2020-08-04 15:28:17 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2020-08-04 15:28:17 2 [Note] WSREP: State transfer required:
        Group state: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1733
        Local state: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731
2020-08-04 15:28:17 2 [Note] WSREP: Server status change connected -> joiner
2020-08-04 15:28:17 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-08-04 15:28:17 0 [Note] WSREP: Joiner monitor thread started to monitor
2020-08-04 15:28:17 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '100.124.161.47' --datadir '/bitnami/mariadb/data/' --defaults-file '/opt/bitnami/mariadb/conf/my.cnf' --parent '1' --binlog 'mysql-bin' --mysqld-args --defaults-file/opt/bitnami/mariadb/conf/my.cnf --basedir/opt/bitnami/mariadb --datadir/bitnami/mariadb/data --socket/opt/bitnami/mariadb/tmp/mysql.sock --pid-file/opt/bitnami/mariadb/tmp/mysqld.pid --wsrep_node_namestg-mariadb-galera-1 --wsrep_node_address100.124.161.47 --wsrep_cluster_namegalera --wsrep_cluster_addressgcomm://stg-mariadb-galera-headless.stg.svc.cluster.local --wsrep_sst_methodmariabackup --wsrep_sst_authmariabackup:TRHCIIKBdlyLg2vMrSqDWE@@'
WSREP_SST: [INFO] Streaming with xbstream (20200804 15:28:18.119)
WSREP_SST: [INFO] Using socat as streamer (20200804 15:28:18.124)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC( ${PIPESTATUS[@]} ) (20200804 15:28:18.219)
2020-08-04 15:28:18 2 [Note] WSREP: Prepared SST request: mariabackup|100.124.161.47:4444/xtrabackup_sst//1
2020-08-04 15:28:18 2 [Note] WSREP: ####### IST uuid:6cf9ed1b-d54b-11ea-8cf7-5fde2825032b f: 1732, l: 1733, STRv: 3
2020-08-04 15:28:18 2 [Note] WSREP: IST receiver addr using tcp://100.124.161.47:4568
2020-08-04 15:28:18 2 [Note] WSREP: Prepared IST receiver for 1732-1733, listening at: tcp://100.124.161.47:4568
2020-08-04 15:28:18 0 [Note] WSREP: Member 0.0 (stg-mariadb-galera-1) requested state transfer from '*any*'. Selected 1.0 (stg-mariadb-galera-0)(SYNCED) as donor.
2020-08-04 15:28:18 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1733)
2020-08-04 15:28:18 2 [Note] WSREP: Requesting state transfer: success, donor: 1
2020-08-04 15:28:19 0 [Note] WSREP: 1.0 (stg-mariadb-galera-0): State transfer to 0.0 (stg-mariadb-galera-1) complete.
2020-08-04 15:28:19 0 [Note] WSREP: Member 1.0 (stg-mariadb-galera-0) synced with group.
WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20200804 15:28:19.374)
WSREP_SST: [INFO] Galera co-ords from recovery: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731 0 (20200804 15:28:19.413)
WSREP_SST: [INFO] Total time on joiner: 0 seconds (20200804 15:28:19.432)
WSREP_SST: [INFO] Removing the sst_in_progress file (20200804 15:28:19.443)
2020-08-04 15:28:19 3 [Note] WSREP: SST received
2020-08-04 15:28:19 3 [Note] WSREP: Server status change joiner -> initializing
2020-08-04 15:28:19 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-08-04 15:28:19 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
2020-08-04 15:28:19 0 [Note] InnoDB: Using Linux native AIO
2020-08-04 15:28:19 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-08-04 15:28:19 0 [Note] InnoDB: Uses event mutexes
2020-08-04 15:28:19 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-08-04 15:28:19 0 [Note] InnoDB: Number of pools: 1
2020-08-04 15:28:19 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-08-04 15:28:19 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2020-08-04 15:28:19 0 [Note] InnoDB: Initializing buffer pool, total size  2G, instances  8, chunk size  128M
2020-08-04 15:28:19 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-04 15:28:19 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-04 15:28:19 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 134217728 bytes
2020-08-04 15:28:19 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 134217728 bytes
2020-08-04 15:28:19 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2020-08-04 15:28:19 0 [Note] InnoDB: New log files created, LSN181125986
2020-08-04 15:28:20 0 [Note] WSREP: (1e4b81b8-bbba, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-08-04 15:28:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-08-04 15:28:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-08-04 15:28:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-08-04 15:28:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-08-04 15:28:20 0 [Note] InnoDB: Waiting for purge to start
2020-08-04 15:28:20 0 [Note] InnoDB: 10.4.13 started; log sequence number 181126156; transaction id 2784
2020-08-04 15:28:20 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2020-08-04 15:28:20 0 [Note] Plugin 'FEEDBACK' is disabled.
/opt/bitnami/mariadb/sbin/mysqld, Version: 10.4.13-MariaDB-log (Source distribution). started with:
Tcp port: 3306  Unix socket: /opt/bitnami/mariadb/tmp/mysql.sock
Time                Id Command  Argument
2020-08-04 15:28:20 0 [Note] Server socket created on IP: '0.0.0.0'.
2020-08-04 15:28:20 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
2020-08-04 15:28:20 0 [Note] WSREP: Server initialized
2020-08-04 15:28:20 0 [Note] WSREP: Server status change initializing -> initialized
2020-08-04 15:28:20 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-08-04 15:28:20 3 [Note] WSREP: Server status change initialized -> joined
2020-08-04 15:28:20 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-08-04 15:28:20 3 [Note] WSREP: Recovered position from storage: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731
2020-08-04 15:28:20 3 [Note] WSREP: Recovered view from SST:
  id: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: -1
  members(2):
        0: ab31edb0-d666-11ea-ab39-de43af719e08, stg-mariadb-galera-0
        1: c788c5e8-d666-11ea-bca6-7ecd4cb5e9aa, stg-mariadb-galera-1

2020-08-04 15:28:20 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-08-04 15:28:20 11 [Note] WSREP: Starting applier thread 11
2020-08-04 15:28:20 12 [Note] WSREP: Recovered cluster id 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b
2020-08-04 15:28:20 13 [Note] WSREP: Starting applier thread 13
2020-08-04 15:28:20 0 [Note] Reading of all Master_info entries succeeded
2020-08-04 15:28:20 0 [Note] Added new Master_info '' to hash table
2020-08-04 15:28:20 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.4.13-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
2020-08-04 15:28:20 15 [Note] WSREP: Starting applier thread 15
2020-08-04 15:28:20 3 [Note] WSREP: SST received: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731
2020-08-04 15:28:20 2 [Note] WSREP: Installed new state from SST: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731
2020-08-04 15:28:20 0 [Note] WSREP: Joiner monitor thread ended with total time 3 sec
2020-08-04 15:28:20 2 [Note] WSREP: Receiving IST: 2 writesets, seqnos 1732-1733
2020-08-04 15:28:20 0 [Note] WSREP: ####### IST applying starts with 1732
2020-08-04 15:28:20 0 [Note] WSREP: ####### IST current seqno initialized to 1732
2020-08-04 15:28:20 0 [Note] WSREP: Receiving IST...  0.0% (0/2 events) complete.
2020-08-04 15:28:20 0 [Note] WSREP: REPL Protocols: 10 (5)
2020-08-04 15:28:20 0 [Note] WSREP: Service thread queue flushed.
2020-08-04 15:28:20 0 [Note] WSREP: ####### Assign initial position for certification: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1731, protocol version: 5
2020-08-04 15:28:20 0 [Note] WSREP: REPL Protocols: 10 (5)
2020-08-04 15:28:20 0 [Note] WSREP: ####### Adjusting cert position: 1731 -> 1732
2020-08-04 15:28:20 0 [Note] WSREP: Service thread queue flushed.
2020-08-04 15:28:20 0 [Note] WSREP: Lowest cert index boundary for CC from ist: 1732
2020-08-04 15:28:20 0 [Note] WSREP: Min available from gcache for CC from ist: 1402
2020-08-04 15:28:20 0 [Note] WSREP: IST preload starting at 1733
2020-08-04 15:28:20 11 [Note] WSREP: 
View:
  id: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1732
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: -1
  members(1):
        0: ab31edb0-d666-11ea-ab39-de43af719e08, stg-mariadb-galera-0

2020-08-04 15:28:20 11 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-08-04 15:28:20 0 [Note] WSREP: REPL Protocols: 10 (5)
2020-08-04 15:28:20 0 [Note] WSREP: ####### Adjusting cert position: 1732 -> 1733
2020-08-04 15:28:20 0 [Note] WSREP: Service thread queue flushed.
2020-08-04 15:28:20 0 [Note] WSREP: Lowest cert index boundary for CC from ist: 1733
2020-08-04 15:28:20 0 [Note] WSREP: Min available from gcache for CC from ist: 1402
2020-08-04 15:28:20 0 [Note] WSREP: Receiving IST...100.0% (2/2 events) complete.
2020-08-04 15:28:20 13 [Note] WSREP: 
View:
  id: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1733
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(2):
        0: 1e4b81b8-d667-11ea-bbba-a6360c66a8a4, stg-mariadb-galera-1
        1: ab31edb0-d666-11ea-ab39-de43af719e08, stg-mariadb-galera-0

2020-08-04 15:28:20 13 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-08-04 15:28:20 2 [Note] WSREP: Draining apply monitors after IST up to 1733
2020-08-04 15:28:20 2 [Note] WSREP: IST received: 6cf9ed1b-d54b-11ea-8cf7-5fde2825032b:1733
2020-08-04 15:28:20 2 [Note] WSREP: Lowest cert index boundary for CC from sst: 1733
2020-08-04 15:28:20 2 [Note] WSREP: Min available from gcache for CC from sst: 1402
2020-08-04 15:28:20 0 [Note] WSREP: 0.0 (stg-mariadb-galera-1): State transfer from 1.0 (stg-mariadb-galera-0) complete.
2020-08-04 15:28:20 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 1733)
2020-08-04 15:28:20 0 [Note] WSREP: Member 0.0 (stg-mariadb-galera-1) synced with group.
2020-08-04 15:28:20 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 1733)
2020-08-04 15:28:20 2 [Note] WSREP: Server stg-mariadb-galera-1 synced with group
2020-08-04 15:28:20 2 [Note] WSREP: Server status change joined -> synced
2020-08-04 15:28:20 2 [Note] WSREP: Synchronized with group, ready for connections
2020-08-04 15:28:20 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-08-04 15:28:20 17 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:28:21 18 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:28:30 19 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:28:31 20 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:28:31 0 [Note] InnoDB: Buffer pool(s) load completed at 200804 15:28:31
2020-08-04 15:28:40 21 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:28:41 22 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:28:46 23 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-08-04 15:28:50 24 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:28:51 25 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:28:56 26 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-08-04 15:29:00 27 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:29:01 28 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:29:06 29 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:29:06 30 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-08-04 15:29:10 31 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:29:11 32 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: YES)
2020-08-04 15:29:16 33 [Warning] Access denied for user 'root'@'localhost' (using password: YES)

mn0o7 commented 4 years ago

Deleting the pvc of that pod and then deleting the pods solved the proble. I would be very happy to understand why this is happening

carrodher commented 4 years ago

Hi, can you provide more information about your use case?

What chart version are you using?
Are you facing this issue on a fresh installation or during an upgrade? If it is
Are you using the default values or some custom values via a values.yaml file or --set flag?

When the chart is installed the first time without setting a password, a random password is generated and stored using secrets in the PVC. If you are upgrading an existing deployment, you need to specify the password values, if not it is going to use the previous one that it's not going to match.

In the same way if it is a new installation but there were a previous one with the same name that is going to use the same PVC, it's possible that the same issue happens because helm delete doesn't remove the PVCs, you need to manually remove them with kubectl delete pvc ...

mn0o7 commented 4 years ago

@carrodher Chart version mariadb-galera-3.1.3 All defaults except for passwords and these : galera.forceSafeToBootstrap: true persistence.storageClass: "rook-ceph-block" extraEnvVars:

name: MARIADB_INIT_SLEEP_TIME value: "300"

carrodher commented 4 years ago

I was not able to reproduce the issue using this configuration, I think the issue is caused because of the random password that needs to be removed or reused in the next deployments:

When the chart is installed the first time without setting a password, a random password is generated and stored using secrets in the PVC. If you are upgrading an existing deployment, you need to specify the password values, if not it is going to use the previous one that it's not going to match.

In the same way if it is a new installation but there was a previous one with the same name that is going to use the same PVC, it's possible that the same issue happens because helm delete doesn't remove the PVCs, you need to manually remove them with kubectl delete pvc ...

bitnami / charts

Use helm install failed #1788