search

SonarSource / docker-sonarqube

:whale: SonarQube in Docker
https://hub.docker.com/_/sonarqube/
GNU Lesser General Public License v3.0
1.38k stars 1.02k forks source link

Sonarqube cache DNS resolutions forever causing failed merge request decorations #457

Closed cykl closed 3 years ago

cykl commented 3 years ago

When our Gitlab instance got a new IP address this week, Sonarqube failed to decorate merge requests until we killed the pod. It seems to be caching DNS resolutions forever. Root cause is likely that Sonarqube uses a security manager but do not set networkaddress.cache.ttl to 0 or a small value in the security policy. JVM default value is -1, ie “cache forever” policy, see https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/doc-files/net-properties.html.

It would be great if official Sonarqube docker images set a sensible networkaddress.cache.ttl. As a temporary workaround, users should be able to set -Dsun.net.inetaddr.ttl which is easier to set than writing a security policy file on disk.

Troubleshooting

Merge request decoration was failing with following stack trace:

java.lang.IllegalStateException: Failed to connect to [redacted]/10.46.0.13:443
        at com.sonarsource.C.D.D.G.A(Unknown Source)
        at com.sonarsource.C.D.D.G.B(Unknown Source)
        at com.sonarsource.C.D.D.B.A(Unknown Source)
        at com.sonarsource.C.D.c.A(Unknown Source)
        at java.base/java.util.Optional.ifPresent(Optional.java:183)
        at com.sonarsource.C.D.c.B(Unknown Source)
        at com.sonarsource.C.D.c.A(Unknown Source)
        at org.sonar.ce.async.SynchronousAsyncExecution.addToQueue(SynchronousAsyncExecution.java:27)
        at com.sonarsource.C.D.c.A(Unknown Source)
        at java.base/java.util.Optional.ifPresent(Optional.java:183)
        at com.sonarsource.C.D.c.finished(Unknown Source)
        at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.executeTask(PostProjectAnalysisTasksExecutor.java:118)
        at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.finished(PostProjectAnalysisTasksExecutor.java:109)
        at org.sonar.ce.task.step.ComputationStepExecutor.executeListener(ComputationStepExecutor.java:91)
        at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:63)
        at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:235)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:217)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:162)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:137)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:89)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.ConnectException: Failed to connect to [redacted]/10.46.0.13:443
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:265)
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183)
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
        at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
        at okhttp3.RealCall.execute(RealCall.java:81)
        ... 31 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
        at java.base/java.net.Socket.connect(Socket.java:609)
        at okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
        ... 49 common frames omitted

Sonarqube thinks that gitlab IP address is 10.46.0.13 but DNS says that IP address is 10.46.0.20

$ kube exec -it [redacted]-sonarqube-689f585b7f-k97v9 -- ping -c 1 [redacted]
Defaulting container name to sonarqube.
PING [redacted] (10.46.0.20): 56 data bytes
64 bytes from 10.46.0.20: seq=0 ttl=63 time=0.617 ms

Environement:

krmichelos commented 3 years ago

We had issues with the work around not seeming to work. We have a webhook for returning scan results to Jenkins. Jenkins is behind a dynamic load balancer and the IPs change on a somewhat regular basis. Whenever the IPs would change all our webhook calls would fails as SonarQube now has an incorrect IP for Jenkins. We resorted to building a custom image where we set networkaddress.cache.ttl to 0 to turn the cache off to avoid these failures.