search

gocd / kubernetes-elastic-agents

Kubernetes Elastic agent plugin for GoCD
https://www.gocd.org
Apache License 2.0
34 stars 32 forks source link

Failed to handle request cd.go.elastic-agent.create-agent' - 500 Response #73

Closed kaizenlabs closed 5 years ago

kaizenlabs commented 5 years ago
Issue Type

When I kick off a pipeline build, an agent pod is not created by the elastic agent plugin. I checked the logs of the gocd-pod and see a Java runtime error in there:

java.lang.RuntimeException: Interaction with plugin with id 'cd.go.contrib.elasticagent.kubernetes' implementing 'elastic-agent' extension failed while requesting for 'cd.go.elastic-agent.create-agent'. Reason: [The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.create-agent']
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:57)
        at com.thoughtworks.go.plugin.access.elastic.v4.ElasticAgentExtensionV4.createAgent(ElasticAgentExtensionV4.java:101)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:72)
        at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:41)
        at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:32)
        at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:22)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:73)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:53)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failedto handle request cd.go.elastic-agent.create-agent'
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:55)
Environment
Basic environment details

Set up elastic agent plugin according to the documentation on install:

Put in the GoCD server address Put in the correct cluster address Provided the security token Provided the CA certificate without the BEGIN and END Certificate tags Put in the correct namespace (gocd)

Created a basic agent pod using a YAML file, agent is working fine. Tested locally.

Expected Results

Elastic agents pods should be created in gocd namespace

Actual Results

The create agent request fails.

Any other info

Odd thing is that the agent pods can launch when I leave the default credentials in there, but then they don't actually execute any of the tasks or jobs in the pipeline. So I saw that they were working, but when I changed the elastic agent plugin configuration, I get the Java runtime error above.

arvindsv commented 5 years ago

@sheroy @ankitsri11 Can one of you take a look? This issue should probably be in: https://github.com/gocd/kubernetes-elastic-agents/issues

arvindsv commented 5 years ago

You can now transfer issues, between GitHub repositories! Moved it to the elastic agent's issues.

ankitsri11 commented 5 years ago

@JohnAntonusMaximus - In order to troubleshoot this further can you help us with below information:

Can you check to see if you have compatibility issues between the GoCD server and Kubernetes elastic agent. The compatibility requirements are called out in the change log .

childsc commented 5 years ago

Hello @ankitsri11 We are seeing this issue as well. GoCD server version - 19.3.0 Where is GoCD server running? Docker image orchestrated through Rancher Kubernetes-elastic-agent plugin version v3.0.0-156

This is the first error we see, seen earlier than the one posted by @JohnAntonusMaximus 

> 2019-05-06 07:10:05,028 ERROR [Thread-82] p.c.g.c.e.k.c.g.c.e.KubernetesPlugin:128 [plugin-cd.go.contrib.elasticagent.kubernetes] - Failed to handle request go.plugin-settings.get-configuration
> 5/6/2019 9:10:05 AMjava.lang.NullPointerException: null
> 5/6/2019 9:10:05 AM   at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:62)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:153)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:304)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:246)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:150)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.access.common.AbstractExtension.getPluginSettingsConfiguration(AbstractExtension.java:57)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.access.common.settings.PluginSettingsMetadataLoader.lambda$findSettingsAndViewOfAllExtensionsIn$2(PluginSettingsMetadataLoader.java:85)
> 5/6/2019 9:10:05 AM   at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 5/6/2019 9:10:05 AM   at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> 5/6/2019 9:10:05 AM   at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> 5/6/2019 9:10:05 AM   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> 5/6/2019 9:10:05 AM   at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> 5/6/2019 9:10:05 AM   at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> 5/6/2019 9:10:05 AM   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> 5/6/2019 9:10:05 AM   at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.access.common.settings.PluginSettingsMetadataLoader.findSettingsAndViewOfAllExtensionsIn(PluginSettingsMetadataLoader.java:90)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.access.common.settings.PluginSettingsMetadataLoader.fetchPluginSettingsMetaData(PluginSettingsMetadataLoader.java:60)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.access.common.settings.PluginSettingsMetadataLoader.pluginLoaded(PluginSettingsMetadataLoader.java:50)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.lambda$notifyPluginLoadedEvent$2(FelixGoPluginOSGiFramework.java:351)
> 5/6/2019 9:10:05 AM   at org.apache.commons.collections4.IteratorUtils.forEach(IteratorUtils.java:1224)
> 5/6/2019 9:10:05 AM   at org.apache.commons.collections4.IterableUtils.forEach(IterableUtils.java:584)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.getBundle(FelixGoPluginOSGiFramework.java:133)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.loadPlugin(FelixGoPluginOSGiFramework.java:103)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.listeners.DefaultPluginJarChangeListener.refreshBundle(DefaultPluginJarChangeListener.java:151)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.listeners.DefaultPluginJarChangeListener.addPlugin(DefaultPluginJarChangeListener.java:101)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.listeners.DefaultPluginJarChangeListener.pluginJarAdded(DefaultPluginJarChangeListener.java:66)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread$DoOnAllListeners.lambda$pluginJarAdded$0(DefaultPluginJarLocationMonitor.java:283)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread$DoOnAllListeners.doOnAllPluginJarChangeListener(DefaultPluginJarLocationMonitor.java:304)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread$DoOnAllListeners.pluginJarAdded(DefaultPluginJarLocationMonitor.java:283)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread.notifyListenersOfAddedPlugins(DefaultPluginJarLocationMonitor.java:217)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread.loadAndNotifyPluginsFrom(DefaultPluginJarLocationMonitor.java:208)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread.oneShot(DefaultPluginJarLocationMonitor.java:196)
> 5/6/2019 9:10:05 AM   at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread.run(DefaultPluginJarLocationMonitor.java:181)
> 5/6/2019 9:10:05 AM2019-05-06 07:10:05,029 WARN  [Thread-82] PluginSettingsMetadataLoader:64 - Failed to fetch plugin settings metadata for plugin cd.go.contrib.elasticagent.kubernetes. Maybe the plugin does not implement plugin settings and view?
> 5/6/2019 9:10:05 AM2019-05-06 07:10:05,031 WARN  [Thread-82] PluginSettingsMetadataLoader:65 - Plugin: cd.go.contrib.elasticagent.kubernetes - Metadata load info: [{extension='elastic-agent', configuration='null', view='null', error='The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request go.plugin-settings.get-configuration''}]
> 5/6/2019 9:10:05 AM2019-05-06 07:10:05,032 WARN  [Thread-82] PluginSettingsMetadataLoader:66 - Not all plugins are required to implement the request above. This error may be safe to ignore.
ankitsri11 commented 5 years ago

@childsc As part of GoCD version v19.3.0, we've introduced multi-cluster support and removed support for plugin-settings. So the error we're seeing here is safe to ignore.

For the original issue, it looks like misconfiguration issue. We might be able to answer more if you can share the below information:

  1. GoCD server logs (When the job is scheduled)
  2. Entire plugin logs
  3. (if possible) the cluster profile and elastic agent profile configurations

I would suggest sharing the above information on our support channel (support@thoughtworks.com). Please add the issue number and someone from the team will take a look. Thanks.

arvindsv commented 5 years ago

Maybe @bdpiparva or @GaneshSPatil knows whether that call to get plugin settings was supposed to be made, in this case. Since it's a new server and a new plugin.

GaneshSPatil commented 5 years ago

@arvindsv -- The call to get the plugin settings still happen for the v5 plugin. It's a generic get-plugin-settings call made for all the plugins, regardless of plugin supporting plugin-settings or not.

As @ankitsri11 mentioned, this error is safe to ignore.

bdpiprava commented 5 years ago

@GaneshSPatil - IIRC, we decided that if the plugin is a type of elastic agent then we will not make the plugin settings related calls.

Why

If the plugin uses the plugin settings then on a load of the plugin the settings will be migrated to cluster profile so making plugin related calls are not right.

nitingadekar commented 5 years ago

I am trying to create a pipeline, for which I created a Elastic Profile. I am getting error when I tried to generate status report as below:

Failed to load status report The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.cluster-status-report'

Because of which the job is not getting linked with the profile and the pipeline execution fails. Screenshot from 2019-06-17 15-05-54 Screenshot from 2019-06-17 15-06-45

GaneshSPatil commented 5 years ago

Hi Nitin,

Can you attach the plugin-cd.go.contrib.elasticagent.kubernetes.log file contents to see if there are any errors while loading the status report page?

Note: Please make sure that we strip out sensitive information from the logs.

nitingadekar commented 5 years ago

plugin-cd.go.contrib.elasticagent.kubernetes.log Hi, please check the logs.

nitingadekar commented 5 years ago

Below is exact error log when I hit Status_Report. 

2019-06-17 17:26:15,735 ERROR [159@MessageListener for CreateAgentListener] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.create-agent
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [kube-system]  failed.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:609)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
    at cd.go.contrib.elasticagent.KubernetesAgentInstances.refreshAll(KubernetesAgentInstances.java:150)
    at cd.go.contrib.elasticagent.KubernetesPlugin.refreshInstancesForCluster(KubernetesPlugin.java:134)
    at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:82)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:152)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:303)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:149)
    at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
    at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:142)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:60)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:43)
    at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:32)
    at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:22)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:86)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:66)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:299)
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:268)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:160)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:54)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
    at okhttp3.RealCall.execute(RealCall.java:77)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:327)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:605)
    ... 17 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
    at sun.security.validator.Validator.validate(Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
    ... 53 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
    ... 59 common frames omitted
2019-06-17 17:26:15,744 ERROR [158@MessageListener for CreateAgentListener] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.create-agent
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [kube-system]  failed.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:609)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
    at cd.go.contrib.elasticagent.KubernetesAgentInstances.refreshAll(KubernetesAgentInstances.java:150)
    at cd.go.contrib.elasticagent.KubernetesPlugin.refreshInstancesForCluster(KubernetesPlugin.java:134)
    at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:82)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:152)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:303)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:149)
    at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
    at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:142)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:60)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:43)
    at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:32)
    at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:22)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:86)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:66)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:299)
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:268)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:160)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:54)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
    at okhttp3.RealCall.execute(RealCall.java:77)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:327)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:605)
    ... 17 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
    at sun.security.validator.Validator.validate(Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
    ... 53 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
    ... 59 common frames omitted
2019-06-17 17:26:18,903 ERROR [qtp1684890795-15] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.cluster-status-report
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [kube-system]  failed.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:609)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
    at cd.go.contrib.elasticagent.KubernetesAgentInstances.refreshAll(KubernetesAgentInstances.java:150)
    at cd.go.contrib.elasticagent.KubernetesPlugin.refreshInstancesForCluster(KubernetesPlugin.java:134)
    at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:102)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:152)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:303)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:149)
    at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
    at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.getClusterStatusReport(ElasticAgentExtensionV5.java:207)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.getClusterStatusReport(ElasticAgentExtension.java:108)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.getClusterStatusReport(ElasticAgentPluginRegistry.java:79)
    at com.thoughtworks.go.server.service.ElasticAgentPluginService.getClusterStatusReport(ElasticAgentPluginService.java:237)
    at sun.reflect.GeneratedMethodAccessor309.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:470)
    at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:328)
    at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:57)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:199)
    at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)
    at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:103)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)
    at org.jruby.RubyClass.finvoke(RubyClass.java:525)
    at org.jruby.RubyBasicObject.send(RubyBasicObject.java:1752)
    at org.jruby.RubyKernel.send(RubyKernel.java:2182)
    at org.jruby.RubyKernel$INVOKER$s$send.call(RubyKernel$INVOKER$s$send.gen)
    at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:105)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.basic_implicit_render.invokeSuper2:-unknown-super-target-(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/basic_implicit_render.rb:6)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.basic_implicit_render.RUBY$method$send_action$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/basic_implicit_render.rb:6)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:176)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:72)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.base.invokeOther0:send_action(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/base.rb:194)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.base.RUBY$method$process_action$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/base.rb:194)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.rendering.invokeSuper6:-unknown-super-target-(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/rendering.rb:30)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.rendering.RUBY$method$process_action$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/rendering.rb:30)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
    at org.jruby.ir.runtime.IRRuntimeHelpers.zSuperSplatArgs(IRRuntimeHelpers.java:1127)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.callbacks.invokeSuper2:-unknown-super-target-(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/callbacks.rb:42)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.callbacks.RUBY$block$process_action$1(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/callbacks.rb:42)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
    at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:85)
    at org.jruby.runtime.Block.yieldSpecific(Block.java:134)
    at org.jruby.ir.runtime.IRRuntimeHelpers.yieldSpecific(IRRuntimeHelpers.java:463)
    at org.jruby.ir.targets.YieldSite.yieldSpecific(YieldSite.java:119)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.callbacks.RUBY$method$run_callbacks$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/callbacks.rb:132)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:178)
    at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:187)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.callbacks.invokeOther6:run_callbacks(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/callbacks.rb:41)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.callbacks.RUBY$method$process_action$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/callbacks.rb:41)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.rescue.invokeSuper0:-unknown-super-target-(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/rescue.rb:22)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.rescue.RUBY$method$process_action$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/rescue.rb:22)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
    at org.jruby.ir.runtime.IRRuntimeHelpers.zSuperSplatArgs(IRRuntimeHelpers.java:1127)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.instrumentation.invokeSuper3:-unknown-super-target-(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/instrumentation.rb:38)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.instrumentation.RUBY$block$process_action$1(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/instrumentation.rb:38)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
    at org.jruby.runtime.Block.yield(Block.java:165)
    at org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:459)
    at org.jruby.ir.targets.YieldSite.yield(YieldSite.java:92)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.notifications.RUBY$block$instrument$1(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/notifications.rb:168)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
    at org.jruby.runtime.Block.yield(Block.java:165)
    at org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:459)
    at org.jruby.ir.targets.YieldSite.yield(YieldSite.java:92)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.notifications.instrumenter.RUBY$method$instrument$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/notifications/instrumenter.rb:23)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:111)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:187)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:209)
    at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:218)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.notifications.invokeOther5:instrument(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/notifications.rb:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.notifications.RUBY$method$instrument$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/notifications.rb:168)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:111)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:187)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:209)
    at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:218)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.instrumentation.invokeOther43:instrument(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/instrumentation.rb:32)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.instrumentation.RUBY$method$process_action$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/instrumentation.rb:32)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.params_wrapper.invokeSuper20:-unknown-super-target-(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/params_wrapper.rb:256)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.params_wrapper.RUBY$method$process_action$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal/params_wrapper.rb:256)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:176)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.base.invokeOther11:process_action(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/base.rb:134)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.abstract_controller.base.RUBY$method$process$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/abstract_controller/base.rb:134)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1119)
    at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuperSplatArgs(IRRuntimeHelpers.java:1101)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionview_minus_5_dot_2_dot_2_dot_1.lib.action_view.rendering.invokeSuper5:-unknown-super-target-(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionview-5.2.2.1/lib/action_view/rendering.rb:32)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionview_minus_5_dot_2_dot_2_dot_1.lib.action_view.rendering.RUBY$method$process$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionview-5.2.2.1/lib/action_view/rendering.rb:32)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:98)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.invokeOther2:process(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal.rb:191)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.RUBY$method$dispatch$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal.rb:191)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:127)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:219)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:230)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.invokeOther10:dispatch(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal.rb:252)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_controller.metal.RUBY$method$dispatch$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_controller/metal.rb:252)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:127)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:219)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:230)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.invokeOther0:dispatch(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:52)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.RUBY$method$dispatch$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:52)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.RUBY$method$dispatch$0$__VARARGS__(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:88)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:176)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.invokeOther6:dispatch(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:34)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.RUBY$method$serve$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:34)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.journey.router.invokeOther27:serve(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/journey/router.rb:52)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.journey.router.RUBY$block$serve$1(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/journey/router.rb:52)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
    at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
    at org.jruby.runtime.Block.yield(Block.java:165)
    at org.jruby.RubyArray.each(RubyArray.java:1801)
    at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:147)
    at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:156)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.journey.router.invokeOther37:each(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/journey/router.rb:35)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.journey.router.RUBY$method$serve$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/journey/router.rb:35)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.invokeOther5:serve(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:840)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.routing.route_set.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/routing/route_set.rb:840)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.versionist_minus_1_dot_7_dot_0.lib.versionist.middleware.invokeOther63:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/versionist-1.7.0/lib/versionist/middleware.rb:39)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.versionist_minus_1_dot_7_dot_0.lib.versionist.middleware.RUBY$method$_call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/versionist-1.7.0/lib/versionist/middleware.rb:39)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.versionist_minus_1_dot_7_dot_0.lib.versionist.middleware.invokeOther1:_call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/versionist-1.7.0/lib/versionist/middleware.rb:17)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.versionist_minus_1_dot_7_dot_0.lib.versionist.middleware.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/versionist-1.7.0/lib/versionist/middleware.rb:17)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.tempfile_reaper.invokeOther8:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.tempfile_reaper.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.etag.invokeOther5:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/etag.rb:25)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.etag.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/etag.rb:25)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.conditional_get.invokeOther26:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:25)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.conditional_get.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:25)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.head.invokeOther5:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.head.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.http.content_security_policy.invokeOther2:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/http/content_security_policy.rb:18)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.http.content_security_policy.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/http/content_security_policy.rb:18)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at uri_3a_classloader_3a_.jruby.rack.session_store.invokeOther2:call(uri:classloader:/jruby/rack/session_store.rb:79)
    at uri_3a_classloader_3a_.jruby.rack.session_store.RUBY$method$context$0(uri:classloader:/jruby/rack/session_store.rb:79)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:75)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:98)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.session.abstract.id.invokeOther0:context(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.session.abstract.id.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.cookies.invokeOther2:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/cookies.rb:670)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.cookies.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/cookies.rb:670)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.callbacks.invokeOther1:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/callbacks.rb:28)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.callbacks.RUBY$block$call$1(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/callbacks.rb:28)
    at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:162)
    at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:85)
    at org.jruby.runtime.Block.yieldSpecific(Block.java:134)
    at org.jruby.ir.runtime.IRRuntimeHelpers.yieldSpecific(IRRuntimeHelpers.java:463)
    at org.jruby.ir.targets.YieldSite.yieldSpecific(YieldSite.java:119)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.callbacks.RUBY$method$run_callbacks$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/callbacks.rb:98)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:178)
    at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:187)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.callbacks.invokeOther5:run_callbacks(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/callbacks.rb:26)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.callbacks.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/callbacks.rb:26)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.debug_exceptions.invokeOther2:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.debug_exceptions.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.show_exceptions.invokeOther2:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.show_exceptions.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.rack.logger.invokeOther11:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/rack/logger.rb:38)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.rack.logger.RUBY$method$call_app$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/rack/logger.rb:38)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:114)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:187)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:199)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.rack.logger.invokeOther10:call_app(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/rack/logger.rb:28)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.rack.logger.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/rack/logger.rb:28)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.remote_ip.invokeOther7:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/remote_ip.rb:81)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.remote_ip.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/remote_ip.rb:81)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.request_id.invokeOther7:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/request_id.rb:27)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.request_id.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/request_id.rb:27)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.method_override.invokeOther9:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.method_override.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.runtime.invokeOther2:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/runtime.rb:22)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.runtime.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/runtime.rb:22)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.cache.strategy.local_cache_middleware.invokeOther6:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.activesupport_minus_5_dot_2_dot_2_dot_1.lib.active_support.cache.strategy.local_cache_middleware.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/activesupport-5.2.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.executor.invokeOther4:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/executor.rb:14)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.executor.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/executor.rb:14)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.static.invokeOther13:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/static.rb:127)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.actionpack_minus_5_dot_2_dot_2_dot_1.lib.action_dispatch.middleware.static.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/actionpack-5.2.2.1/lib/action_dispatch/middleware/static.rb:127)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.sendfile.invokeOther13:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.rack_minus_2_dot_0_dot_6.lib.rack.sendfile.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.engine.invokeOther3:call(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/engine.rb:524)
    at var.lib.go_minus_server.work.jetty_minus_0_dot_0_dot_0_dot_0_minus_8153_minus_cruise_dot_war_minus__go_minus_any_minus_.webapp.WEB_minus_INF.rails.gems.jruby.$2_dot_5_dot_0.gems.railties_minus_5_dot_2_dot_2_dot_1.lib.rails.engine.RUBY$method$call$0(/var/lib/go-server/work/jetty-0.0.0.0-8153-cruise.war-_go-any-/webapp/WEB-INF/rails/gems/jruby/2.5.0/gems/railties-5.2.2.1/lib/rails/engine.rb:524)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
    at uri_3a_classloader_3a_.rack.handler.servlet.invokeOther4:call(uri:classloader:/rack/handler/servlet.rb:22)
    at uri_3a_classloader_3a_.rack.handler.servlet.RUBY$method$call$0(uri:classloader:/rack/handler/servlet.rb:22)
    at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:101)
    at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:154)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
    at org.jruby.RubyClass.finvoke(RubyClass.java:911)
    at org.jruby.runtime.Helpers.invoke(Helpers.java:356)
    at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:369)
    at org.jruby.javasupport.JavaEmbedUtils$1.callMethod(JavaEmbedUtils.java:117)
    at org.jruby.rack.DefaultRackApplication.call(DefaultRackApplication.java:64)
    at org.jruby.rack.AbstractRackDispatcher.process(AbstractRackDispatcher.java:33)
    at org.jruby.rack.AbstractServlet.service(AbstractServlet.java:34)
    at org.jruby.rack.AbstractServlet.service(AbstractServlet.java:40)
    at com.thoughtworks.go.rackhack.DelegatingServlet.service(DelegatingServlet.java:47)
    at com.thoughtworks.go.rackhack.DelegatingServlet.service(DelegatingServlet.java:52)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:614)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:227)
    at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:78)
    at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:213)
    at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171)
    at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
    at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
    at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
    at com.thoughtworks.go.server.web.FlashLoadingFilter.doFilterInternal(FlashLoadingFilter.java:38)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
    at com.thoughtworks.go.server.newsecurity.filters.DenyIfRefererIsNotFilesFilter.doFilterInternal(DenyIfRefererIsNotFilesFilter.java:52)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
    at com.thoughtworks.go.server.newsecurity.filters.VerifyAuthorityFilter.doFilterInternal(VerifyAuthorityFilter.java:57)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
    at com.thoughtworks.go.server.newsecurity.filters.AbstractUserEnabledCheckFilter.doFilterInternal(AbstractUserEnabledCheckFilter.java:55)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at com.thoughtworks.go.server.newsecurity.filters.ThreadLocalUserFilter.doFilterInternal(ThreadLocalUserFilter.java:41)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
    at com.thoughtworks.go.server.newsecurity.filters.AbstractBasicAuthenticationFilter.filterWhenSecurityDisabled(AbstractBasicAuthenticationFilter.java:112)
    at com.thoughtworks.go.server.newsecurity.filters.AbstractBasicAuthenticationFilter.doFilterInternal(AbstractBasicAuthenticationFilter.java:70)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at com.thoughtworks.go.server.newsecurity.filters.AssumeAnonymousUserFilter.doFilterInternal(AssumeAnonymousUserFilter.java:64)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at com.thoughtworks.go.server.newsecurity.filters.AbstractReAuthenticationFilter.doFilterInternal(AbstractReAuthenticationFilter.java:66)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at com.thoughtworks.go.server.newsecurity.filters.InvalidateAuthenticationOnSecurityConfigChangeFilter.doFilterInternal(InvalidateAuthenticationOnSecurityConfigChangeFilter.java:105)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
    at com.thoughtworks.go.server.newsecurity.filters.RememberLastRequestUrlFilter.doFilterInternal(RememberLastRequestUrlFilter.java:38)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
    at com.thoughtworks.go.server.newsecurity.filters.AlwaysCreateSessionFilter.doFilterInternal(AlwaysCreateSessionFilter.java:40)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at com.thoughtworks.go.server.newsecurity.filters.ModeAwareFilter.doFilter(ModeAwareFilter.java:80)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
    at com.thoughtworks.go.server.newsecurity.filterchains.MainFilterChain.doFilter(MainFilterChain.java:76)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at com.thoughtworks.go.server.web.BackupFilter.doFilter(BackupFilter.java:89)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at com.thoughtworks.go.server.web.DefaultHeadersFilter.doFilter(DefaultHeadersFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:502)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:299)
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:268)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:160)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:54)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
    at okhttp3.RealCall.execute(RealCall.java:77)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:327)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:605)
    ... 473 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
    at sun.security.validator.Validator.validate(Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
    ... 509 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
    ... 515 common frames omitted
2019-06-17 17:26:43,849 ERROR [164@MessageListener for ServerPingListener] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.server-ping
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [kube-system]  failed.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:609)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
    at cd.go.contrib.elasticagent.KubernetesAgentInstances.refreshAll(KubernetesAgentInstances.java:150)
    at cd.go.contrib.elasticagent.KubernetesPlugin.refreshInstancesForCluster(KubernetesPlugin.java:134)
    at cd.go.contrib.elasticagent.KubernetesPlugin.refreshInstancesForAllClusters(KubernetesPlugin.java:124)
    at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:92)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:152)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:303)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:149)
    at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
    at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.serverPing(ElasticAgentExtensionV5.java:152)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.serverPing(ElasticAgentExtension.java:64)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.serverPing(ElasticAgentPluginRegistry.java:52)
    at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:31)
    at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:21)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:86)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:66)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:299)
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:268)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:160)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:54)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
    at okhttp3.RealCall.execute(RealCall.java:77)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:327)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:605)
    ... 18 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
    at sun.security.validator.Validator.validate(Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
    ... 54 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
    ... 60 common frames omitted
GaneshSPatil commented 5 years ago

@nitingadekar -- This generally happens when an invalid CA certificate is configured. I suggest you to verify your security_token and kubernetes_cluster_ca_cert configured as part of the cluster profile.

nitingadekar commented 5 years ago

@GaneshSPatil I have corrected the token and ca_cert, still getting below error in plugin-cd.go.contrib.elasticagent.kubernetes.log

Message: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" in the namespace "default".

2019-06-18 18:03:46,841 ERROR [qtp1684890795-28] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.cluster-status-report
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://api-kubeaws-k8s-local-oc2hi0-2064939130.ap-south-1.elb.amazonaws.com/api/v1/namespaces/default/pods. Message: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" in the namespace "default". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" in the namespace "default", metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:470)

Is there need to create a custom Service account with custom role assigned to it?

varshavaradarajan commented 5 years ago

@nitingadekar - As mentioned in the security token section of https://github.com/gocd/kubernetes-elastic-agents/blob/master/install.md#configure-cluster-profile , the SA whose token you're using must have the following permissions:

- nodes: list, get
- events: list, watch
- namespace: list, get
- pods, pods/log: *
nitingadekar commented 5 years ago

@GaneshSPatil @varshavaradarajan Thanks for your help, issue resolved after giving mentioned permissions, along with get, list permission to secrets.