search

apache / solr-operator

Official Kubernetes operator for Apache Solr
https://solr.apache.org/operator
Apache License 2.0
243 stars 112 forks source link

While Enabling Javaagent For Newrelic In Apache Solr v9.2 Pods Are Not Running #573

Closed pavanKumar2211 closed 1 year ago

pavanKumar2211 commented 1 year ago

We are deploying apache solr operator https://github.com/apache/solr-operator in AWS EKS 1.23 via helm Currently we are using: Apache Solr operator Image: v0.5.1 Apache Solrcloud Image: v8.8

Now we are upgrading to 9.2 which is compatible with operator 0.5.1

We are installing newrelic agent by downloading newrelic.jar file from https://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip

Under solrOptions to execute newrelic jar we are giving javaOpts="-javaagent:/path/to/newrelic.jar" for solr 8.8. Now we are upgrading to 9.2 which is compatible with operator 0.5.1. With the solrOptions in solr v8.8 the apache-solrcloud pods are running fine and solr is being traced over newrelic but using same configuration for solr v9.2 apache-solrcloud pods are failing to start.

Attaching complete error stacktrace

We are getting below error:

Starting Solr Java 17 detected. Enabled workaround for SOLR-16463 [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true WARNING: A command line option has enabled the Security Manager WARNING: The Security Manager is deprecated and will be removed in a future release OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended 2023-05-26T10:06:38,650+0000 [??? 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/opt/solr-9.2.1/fnp-libs/./newrelic.yml" 2023-05-26T10:06:38,731+0000 [??? 1] com.newrelic INFO: Using default collector host: collector.newrelic.com 2023-05-26T10:06:38,731+0000 [??? 1] com.newrelic INFO: Using default metric ingest URI: https://metric-api.newrelic.com/metric/v1 2023-05-26T10:06:38,732+0000 [??? 1] com.newrelic INFO: Using default event ingest URI: https://insights-collector.newrelic.com/v1/accounts/events 2023-05-26T10:06:38,762+0000 [??? 1] com.newrelic WARN: Error parsing labels - empty name or value 2023-05-26T10:06:38,762+0000 [??? 1] com.newrelic WARN: Labels will not be sent to New Relic 2023-05-26T10:06:38,816+0000 [??? 1] com.newrelic INFO: New Relic Agent: Writing to log file: /opt/solr-9.2.1/fnp-libs/logs/newrelic_agent.log Unable to start New Relic Agent. Please remove -javaagent from your startup arguments and contact New Relic support. java.security.AccessControlException: access denied ("java.io.FilePermission" "/proc/sys/kernel/random/boot_id" "read") at java.base/java.security.AccessControlContext.checkPermission(Unknown Source) at java.base/java.security.AccessController.checkPermission(Unknown Source) at java.base/java.lang.SecurityManager.checkPermission(Unknown Source) at java.base/java.lang.SecurityManager.checkRead(Unknown Source) at java.base/java.io.FileInputStream.(Unknown Source) at com.newrelic.agent.deps.com.google.common.io.Files$FileByteSource.openStream(Files.java:132) at com.newrelic.agent.deps.com.google.common.io.Files$FileByteSource.read(Files.java:156) at com.newrelic.agent.deps.com.google.common.io.ByteSource$AsCharSource.read(ByteSource.java:486) at com.newrelic.agent.utilization.DataFetcher.getBootId(DataFetcher.java:121) at com.newrelic.agent.utilization.UtilizationService.(UtilizationService.java:94) at com.newrelic.agent.service.ServiceManagerImpl.doStart(ServiceManagerImpl.java:225) at com.newrelic.agent.service.AbstractService.start(AbstractService.java:63) at com.newrelic.agent.Agent.continuePremain(Agent.java:166) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:172) at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:114) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source) Error bootstrapping New Relic agent: java.lang.reflect.InvocationTargetException java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:172) at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:114) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source) Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "exitVM.1") at java.base/java.security.AccessControlContext.checkPermission(Unknown Source) at java.base/java.security.AccessController.checkPermission(Unknown Source) at java.base/java.lang.SecurityManager.checkPermission(Unknown Source) at java.base/java.lang.SecurityManager.checkExit(Unknown Source) at java.base/java.lang.Runtime.exit(Unknown Source) at java.base/java.lang.System.exit(Unknown Source) at com.newrelic.agent.Agent.continuePremain(Agent.java:222) ... 12 more 2023-05-26 10:06:42.659 INFO (main) [ ] o.e.j.s.Server jetty-10.0.13; built: 2022-12-07T20:13:20.134Z; git: 1c2636ea05c0ca8de1ffd6ca7f3a98ac084c766d; jvm 17.0.6+10 2023-05-26 10:06:42.699 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///opt/solr-9.2.1/server/contexts/] 2023-05-26 10:06:43.152 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet 2023-05-26 10:06:43.168 WARN (main) [ ] o.e.j.w.WebAppContext Failed startup of context o.e.j.w.WebAppContext@8432469{solr-jetty-context.xml,/solr,file:///opt/solr-9.2.1/server/solr-webapp/webapp/,UNAVAILABLE}{/opt/solr-9.2.1/server/solr-webapp/webapp} => java.lang.VerifyError: Bad type on operand stack Exception Details: java.lang.VerifyError: Bad type on operand stack Exception Details: Location: org/eclipse/jetty/servlet/listener/ELContextCleaner.contextDestroyed(Ljavax/servlet/ServletContextEvent;)V @157: invokeinterface Reason: Type 'java/lang/Object' (current frame, stack[2]) is not assignable to 'java/lang/Throwable' Current Frame: bci: @157 flags: { } locals: { 'org/eclipse/jetty/servlet/listener/ELContextCleaner', 'javax/servlet/ServletContextEvent', 'com/newrelic/agent/bridge/ExitTracer', integer, top, 'javax/servlet/ServletContextEvent', 'org/eclipse/jetty/servlet/listener/ELContextCleaner', 'java/lang/Object' } stack: { 'org/slf4j/Logger', 'java/lang/String', 'java/lang/Object' } Bytecode: 0000000: 014d b200 272a 1007 0110 0eb9 003b 0500 0000010: 4da7 0004 5703 3eb8 003f b900 4501 00b2 0000020: 004b 0312 4d05 bd00 4f59 032a b600 53b6 0000030: 0059 5359 0412 7b53 b900 6005 0057 a700 0000040: 163a 04b2 0027 1904 1228 b900 2e03 00a7 0000050: 0003 043e 2a2b 3a05 3a06 127d b800 833a 0000060: 0719 0619 07b6 0087 3a08 1908 04b6 008d 0000070: 1906 1908 b600 91b2 0093 b900 9901 0099 0000080: 000d b200 9312 9bb9 009f 0200 a700 253a 0000090: 07a7 0020 3a07 b200 9312 a119 07b9 00a5 00000a0: 0300 a700 0f3a 07b2 0093 12a7 b900 9f02 00000b0: 00a7 0003 031d 9f00 122c c600 0d2c 1100 00000c0: b101 b900 6603 00b1 2cc6 000d 2c11 00b1 00000d0: 01b9 0066 0300 b13a 04b2 0027 1904 1228 00000e0: b900 2e03 00a7 0003 2cc6 000d 2c11 00b1 00000f0: 01b9 0066 0300 b12c c600 0b59 2c5f b900 0000100: 6902 00bf
Exception Handler Table: bci [2, 21] => handler: 20 bci [90, 140] => handler: 143 bci [90, 140] => handler: 148 bci [90, 140] => handler: 148 bci [90, 140] => handler: 148 bci [90, 140] => handler: 165 bci [23, 65] => handler: 65 bci [200, 215] => handler: 215 bci [2, 247] => handler: 247 Stackmap Table: full_frame(@20,{Object[#3],Object[#111],Object[#98]},{Object[#31]}) same_frame(@21) full_frame(@65,{Object[#3],Object[#111],Object[#98],Integer},{Object[#31]}) append_frame(@82,Object[#31]) chop_frame(@84,1) full_frame(@140,{Object[#3],Object[#111],Object[#98],Integer,Top,Object[#111],Object[#3],Object[#85],Object[#137]},{}) full_frame(@143,{Object[#3],Object[#111],Object[#98],Integer,Top,Object[#111],Object[#3]},{Object[#114]}) same_locals_1_stack_item_frame(@148,Object[#5]) same_locals_1_stack_item_frame(@165,Object[#122]) append_frame(@177,Object[#5]) same_frame(@180) same_frame(@199) same_frame(@200) same_frame(@214) same_locals_1_stack_item_frame(@215,Object[#31]) full_frame(@232,{Object[#3],Object[#111],Object[#98],Integer,Object[#31],Object[#111],Object[#3],Object[#5]},{}) same_frame(@246) full_frame(@247,{Object[#3],Object[#111],Object[#98]},{Object[#31]}) same_locals_1_stack_item_frame(@259,Object[#31])

at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?]
at java.lang.Class.privateGetDeclaredConstructors(Unknown Source) ~[?:?]
at java.lang.Class.getConstructor0(Unknown Source) ~[?:?]
at java.lang.Class.getDeclaredConstructor(Unknown Source) ~[?:?]
at org.eclipse.jetty.server.handler.ContextHandler$StaticContext.createInstance(ContextHandler.java:2896) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ServletContextHandler$Context.createInstance(ServletContextHandler.java:1292) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ServletContextHandler$Context.createInstance(ServletContextHandler.java:1301) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.BaseHolder.createInstance(BaseHolder.java:204) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ListenerHolder.createInstance(ListenerHolder.java:100) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ListenerHolder.doStart(ListenerHolder.java:89) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:382) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1304) ~[jetty-webapp-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:900) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306) ~[jetty-servlet-10.0.13.jar:10.0.13]
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:532) ~[jetty-webapp-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:221) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.Server.start(Server.java:470) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.server.Server.doStart(Server.java:415) ~[jetty-server-10.0.13.jar:10.0.13]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93) ~[jetty-util-10.0.13.jar:10.0.13]
at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$4(XmlConfiguration.java:1880) ~[jetty-xml-10.0.13.jar:10.0.13]
at java.security.AccessController.doPrivileged(Unknown Source) ~[?:?]
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1818) ~[jetty-xml-10.0.13.jar:10.0.13]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.eclipse.jetty.start.Main.invokeMain(Main.java:229) ~[start.jar:10.0.13]
at org.eclipse.jetty.start.Main.start(Main.java:527) ~[start.jar:10.0.13]
at org.eclipse.jetty.start.Main.main(Main.java:76) ~[start.jar:10.0.13]

2023-05-26 10:06:43.194 INFO (main) [ ] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2023_05_26.request.log 2023-05-26 10:06:43.215 INFO (main) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2017f6e6{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983} 2023-05-26 10:06:43.219 INFO (main) [ ] o.e.j.s.Server Started Server@3db6dd52{STARTING}[10.0.13,sto=0] @5552ms 2023-05-26 10:06:56.685 WARN (qtp718057154-60) [ ] o.e.j.i.ManagedSelector Could not accept java.nio.channels.SocketChannel[closed]: java.lang.VerifyError: Bad type on operand stack

HoustonPutman commented 1 year ago

So the issue is that Solr 9.x has the Java Security Manager enabled by default, while Solr 8.x did not.

It looks like the Newrelic jar you are using requires permissions that do not come out-of-the-box with Solr, which is unsurprising. You can provide a custom envVar SOLR_SECURITY_MANAGER_ENABLED=false to disable the Java security manager.

Unfortunately, I don't think there is a very good way to upload a custom security manager via the operator, unless you want to make your own Solr image that pulls the official image, and overrides the security manager config.

pavanKumar2211 commented 1 year ago

So the issue is that Solr 9.x has the Java Security Manager enabled by default, while Solr 8.x did not.

It looks like the Newrelic jar you are using requires permissions that do not come out-of-the-box with Solr, which is unsurprising. You can provide a custom envVar SOLR_SECURITY_MANAGER_ENABLED=false to disable the Java security manager.

Unfortunately, I don't think there is a very good way to upload a custom security manager via the operator, unless you want to make your own Solr image that pulls the official image, and overrides the security manager config.

Thanks a lot! I added the envVar in Dockerfile provided by solr and pushed into my repository and used it. It worked.

HoustonPutman commented 1 year ago

You can also provide custom envVars through SolrCloud.spec.customSolrKubeOptions.podOptions.envVars. That's probably an easier solution than creating a custom image đŸ™‚