search

SonarSource / docker-sonarqube

:whale: SonarQube in Docker
https://hub.docker.com/_/sonarqube/
GNU Lesser General Public License v3.0
1.38k stars 1.02k forks source link

Elasticsearch crashing SonarQube in Kubernetes #615

Closed OskarMakowski closed 1 year ago

OskarMakowski commented 1 year ago

I'm trying to deploy SonarQube instance into the Kubernetes on SAP BTP Platform using trial account. I managed to create PostgreSQL instance without a problem. When trying to set up the SonarQube the pod goes into the crashing loop because of an error encountered by ElasticSearch. The log is below:

2023-04-06T09:42:49.570249325Z 2023.04.06 09:42:49 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2023-04-06T09:42:49.606824377Z 2023.04.06 09:42:49 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:33431]
2023-04-06T09:42:49.669780979Z 2023.04.06 09:42:49 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
2023-04-06T09:42:49.689304684Z warning: ignoring JAVA_OPTS=-Duser.timezone=Europe/Warsaw -Xmx2048m; pass JVM parameters via ES_JAVA_OPTS
2023-04-06T09:42:49.691365052Z 2023.04.06 09:42:49 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023-04-06T09:42:51.599218787Z 2023.04.06 09:42:51 ERROR es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
2023-04-06T09:42:51.599512477Z org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/opt/sonarqube/data/es7)
2023-04-06T09:42:51.601587361Z  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.601632103Z  at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.601669018Z  at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.601734124Z  at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.601749504Z  at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.601844171Z  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.601854262Z  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.601906901Z Caused by: java.lang.IllegalStateException: Unable to access 'path.data' (/opt/sonarqube/data/es7)
2023-04-06T09:42:51.602011420Z  at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:66) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602038684Z  at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:226) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602103595Z  at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:168) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602129640Z  at org.elasticsearch.bootstrap.Security.configure(Security.java:115) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602198556Z  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:229) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602208061Z  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602261052Z  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602272504Z  ... 6 more
2023-04-06T09:42:51.602333004Z Caused by: java.nio.file.AccessDeniedException: /opt/sonarqube/data/es7
2023-04-06T09:42:51.602360315Z  at sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]
2023-04-06T09:42:51.602410808Z  at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
2023-04-06T09:42:51.602445696Z  at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
2023-04-06T09:42:51.602498192Z  at sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?]
2023-04-06T09:42:51.602526859Z  at java.nio.file.Files.createDirectory(Unknown Source) ~[?:?]
2023-04-06T09:42:51.602577202Z  at java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?]
2023-04-06T09:42:51.602622396Z  at java.nio.file.Files.createDirectories(Unknown Source) ~[?:?]
2023-04-06T09:42:51.602653646Z  at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:318) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602884428Z  at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:64) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602893143Z  at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:226) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602897181Z  at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:168) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602901088Z  at org.elasticsearch.bootstrap.Security.configure(Security.java:115) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602904222Z  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:229) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602942763Z  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602950063Z  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.8.jar:7.17.8]
2023-04-06T09:42:51.602958203Z  ... 6 more
2023-04-06T09:42:51.603283589Z uncaught exception in thread [main]
2023-04-06T09:42:51.603294522Z java.lang.IllegalStateException: Unable to access 'path.data' (/opt/sonarqube/data/es7)
2023-04-06T09:42:51.603298533Z Likely root cause: java.nio.file.AccessDeniedException: /opt/sonarqube/data/es7
2023-04-06T09:42:51.603302164Z  at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
2023-04-06T09:42:51.603305853Z  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
2023-04-06T09:42:51.603309482Z  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
2023-04-06T09:42:51.607283342Z  at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source)
2023-04-06T09:42:51.607311064Z  at java.base/java.nio.file.Files.createDirectory(Unknown Source)
2023-04-06T09:42:51.607328303Z  at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source)
2023-04-06T09:42:51.607369588Z  at java.base/java.nio.file.Files.createDirectories(Unknown Source)
2023-04-06T09:42:51.607421641Z  at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:318)
2023-04-06T09:42:51.607427464Z  at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:64)
2023-04-06T09:42:51.607453118Z  at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:226)
2023-04-06T09:42:51.607483984Z  at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:168)
2023-04-06T09:42:51.607518618Z  at org.elasticsearch.bootstrap.Security.configure(Security.java:115)
2023-04-06T09:42:51.607531196Z  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:229)
2023-04-06T09:42:51.607587405Z  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
2023-04-06T09:42:51.607596410Z  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
2023-04-06T09:42:51.607716458Z  at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
2023-04-06T09:42:51.607861789Z  at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
2023-04-06T09:42:51.607983912Z  at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
2023-04-06T09:42:51.608027575Z  at org.elasticsearch.cli.Command.main(Command.java:77)
2023-04-06T09:42:51.608055202Z  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
2023-04-06T09:42:51.608086580Z  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
2023-04-06T09:42:51.608116572Z For complete error details, refer to the log at /opt/sonarqube/logs/sonarqube.log
2023-04-06T09:42:51.625584430Z 2023.04.06 09:42:51 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 1
2023-04-06T09:42:51.625932865Z 2023.04.06 09:42:51 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023-04-06T09:42:51.628069990Z 2023.04.06 09:42:51 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
2023-04-06T09:42:51.628081243Z org.elasticsearch.ElasticsearchException: java.lang.InterruptedException
2023-04-06T09:42:51.628084692Z  at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
2023-04-06T09:42:51.628087643Z  at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
2023-04-06T09:42:51.628090520Z  at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
2023-04-06T09:42:51.628093070Z  at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
2023-04-06T09:42:51.628095573Z  at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
2023-04-06T09:42:51.628098040Z  at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
2023-04-06T09:42:51.628100733Z  at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
2023-04-06T09:42:51.628103246Z  at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:84)
2023-04-06T09:42:51.628105736Z  at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
2023-04-06T09:42:51.628108213Z  at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
2023-04-06T09:42:51.628110719Z  at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
2023-04-06T09:42:51.628114156Z Caused by: java.lang.InterruptedException: null
2023-04-06T09:42:51.628118100Z  at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)
2023-04-06T09:42:51.628121730Z  at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:243)
2023-04-06T09:42:51.628125665Z  at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
2023-04-06T09:42:51.628129103Z  at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
2023-04-06T09:42:51.628133712Z  ... 10 common frames omitted
2023-04-06T09:42:51.629405344Z 2023.04.06 09:42:51 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

I tried to make it work with 2 images: sonarqube:latest and sonarqube:lts but both produced the same result. Below are all configuration files. sonarqube-deployment.yml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sonar-config
  namespace: dev
  labels:
    app: sonar
data:
  SONARQUBE_JDBC_URL: "jdbc:postgresql://postgres:5432/postgresdb"
  SONARQUBE_JDBC_USERNAME: "sonar_user"
  SONARQUBE_JDBC_PASSWORD: "S0N4RQUB3"
  JAVA_OPTS: "-Duser.timezone=Europe/Warsaw -Xmx2048m"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sonar
  namespace: dev
  labels:
    app: sonar
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: sonar
  template:
    metadata:
      labels:
        app: sonar
    spec:
      initContainers:
      - name: init
        image: busybox
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
      containers:
      - name: sonarqube
        image: sonarqube:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000
        envFrom:
        - configMapRef:
            name: sonar-config
        volumeMounts:
        - name: app-pvc
          mountPath: "/opt/sonarqube/data/"
          subPath: data
        - name: app-pvc
          mountPath: "/opt/sonarqube/extensions/"
          subPath: extensions
        resources:
          requests:
            memory: "1024Mi"
          limits:
            memory: "2048Mi"
      volumes:
      - name: app-pvc
        persistentVolumeClaim:
          claimName: sonar-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: sonar-svc
  namespace: dev
  labels:
    app: sonar
spec:
  ports:
  - port: 9000
    name: sonar
  selector:
    app: sonar

sonarqube-networking.yml:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: sonar
  namespace: sonar
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: myfancy.domain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: sonar
          servicePort: 9000

sonarqube-pvc.yml:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: sonar-pvc
  namespace: dev
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

I tried to log directly into the pod to grant access to the problematic path manually but personally I believe that it’s not possible due to crashing loop of the pod. I had a look at the dockerfile of SonarQube image(dockerfile) and it looks like the access is being set correctly.

carminevassallo commented 1 year ago

Hi @OskarMakowski,

I'm closing this issue as you were opening one similar in our Forum (which is our preferred way to handle user requests). Let's continue our conversation there :)