search

Oteemo / charts

Helm chart repository
https://oteemo.github.io/charts
MIT License
181 stars 233 forks source link

[SonarQube] Unable to perform Kubernetes rolling updates with EFS storage and ReadWriteMany flag. "Sonarqube pod crashing due node.max_local_storage_nodes ?" #104

Open p0bailey opened 4 years ago

p0bailey commented 4 years ago

Hello there and thanks for your effort with this chart,

I'm running Sonar image: 8.2-developer on top of EKS with persistence enabled and looks like there's quite bit of trouble wit ElasticSearch trying to start. Whenever I do a rolling update deployment I get the errors below.

Same thing happens when I scale up the replica with replicaCount: 3

However using deploymentStrategy: type: Recreate mitigate the error. I have a bit of suspicion that the upcoming pod gets into a bit of race condition with the one terminating.

Anyone else experiencing the same issue?

1) https://github.com/Oteemo/charts/issues/53

k logs -p sonarqube-staging-sonarqube-799f945695-dbrvn 2020.05.06 21:35:58 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp 2020.05.06 21:35:58 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001 2020.05.06 21:35:58 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch 2020.05.06 21:35:58 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running 2020.05.06 21:35:58 INFO app[][o.e.p.PluginsService] no modules loaded 2020.05.06 21:35:58 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. 2020.05.06 21:36:00 WARN es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])? at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.4.jar:6.8.4] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.4.jar:6.8.4] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.4.jar:6.8.4] Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])? at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:300) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.node.Node.(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4] ... 6 more 2020.05.06 21:36:00 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1 2020.05.06 21:36:00 INFO app[][o.s.a.SchedulerImpl] Process[es] is stopped 2020.05.06 21:36:00 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped

p0bailey commented 4 years ago

Hello there,

I have also raised a ticket on https://community.sonarsource.com/t/sonarqube-failing-on-kubernetes-eks-with-persisten-storage-and-rolling-update-deploymentstrategy/24294

macnev2013 commented 3 years ago

I was having the same issue. It happens because multiple pods are trying to connect to the same PVC (which is by default set to ReadWriteOnce).

Changing the deploymentStrategy to this worked for me.

deploymentStrategy:
  rollingUpdate:
    maxSurge: 0%
    maxUnavailable: 100%

OR the other option would be to change persistence.accessMode to ReadWriteMany (Refer this doc )

Hope it helps. Thanks.