search

apache / solr-operator

Official Kubernetes operator for Apache Solr
https://solr.apache.org/operator
Apache License 2.0
242 stars 112 forks source link

prometheus-exporter started throwing different errors after upgrading solr operator to 0.7.0 #664

Closed pareekdevanshu closed 6 months ago

pareekdevanshu commented 7 months ago

We recently upgraded solr operator to 0.7.0 in our environments but after the upgrade we are getting errors from prometheus-exporter.

It starts to throw:

ERROR - 2023-12-02 04:04:16.857; org.apache.solr.prometheus.scraper.SolrScraper; java.lang.NullPointerException java.lang.NullPointerException at java.base/java.net.URLEncoder.encode(URLEncoder.java:224) at java.base/java.net.URLEncoder.encode(URLEncoder.java:196) at org.apache.solr.common.params.SolrParams.toQueryString(SolrParams.java:573) at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:372) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260) at org.apache.solr.prometheus.scraper.SolrScraper.collectResponse(SolrScraper.java:109) at org.apache.solr.prometheus.scraper.SolrScraper.call(SolrScraper.java:74) at org.apache.solr.prometheus.scraper.SolrScraper.call(SolrScraper.java:47) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834

and then throws:

ERROR - 2023-12-02 04:04:27.363; org.apache.solr.prometheus.collector.SolrCollector; Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$131/0x000000010031d440@7a70544c rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@54fac02f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 67] java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$131/0x000000010031d440@7a70544c rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@54fac02f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 67] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:194) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) at org.apache.solr.prometheus.collector.SolrCollector.collect(SolrCollector.java:283) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137) at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22) at io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59) at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77) at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82) at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80) at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:692) at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77) at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:664) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) ERROR - 2023-12-02 04:04:36.704; org.apache.solr.prometheus.collector.SolrCollector; null java.lang.NullPointerException at org.apache.solr.util.DOMUtil.addToNamedList(DOMUtil.java:147) at org.apache.solr.util.DOMUtil.nodesToNamedList(DOMUtil.java:117) at org.apache.solr.util.DOMUtil.childNodesToNamedList(DOMUtil.java:107) at org.apache.solr.util.DOMUtil.addToNamedList(DOMUtil.java:156) at org.apache.solr.util.DOMUtil.nodesToNamedList(DOMUtil.java:117) at org.apache.solr.util.DOMUtil.childNodesToNamedList(DOMUtil.java:107) at org.apache.solr.prometheus.collector.SolrCollector.collect(SolrCollector.java:100) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137) at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22) at io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59) at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77) at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82) at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80) at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:692) at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77) at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:664) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

and then eventually starts throwing stackoverflow error:

Exception in thread "pool-2-thread-4" Exception in thread "pool-2-thread-1" java.lang.StackOverflowError at org.apache.solr.util.DOMUtil.getText(DOMUtil.java:223) at org.apache.solr.util.DOMUtil.getText(DOMUtil.java:237) at org.apache.solr.util.DOMUtil.getText(DOMUtil.java:216) at org.apache.solr.util.DOMUtil.addToNamedList(DOMUtil.java:160) at org.apache.solr.util.DOMUtil.nodesToNamedList(DOMUtil.java:117) at org.apache.solr.util.DOMUtil.childNodesToNamedList(DOMUtil.java:107) at org.apache.solr.util.DOMUtil.addToNamedList(DOMUtil.java:156) at org.apache.solr.util.DOMUtil.nodesToNamedList(DOMUtil.java:117) at org.apache.solr.util.DOMUtil.childNodesToNamedList(DOMUtil.java:107) at org.apache.solr.util.DOMUtil.addToNamedList(DOMUtil.java:156) at org.apache.solr.util.DOMUtil.nodesToNamedList(DOMUtil.java:117) at org.apache.solr.util.DOMUtil.childNodesToNamedList(DOMUtil.java:107) at org.apache.solr.util.DOMUtil.addToNamedList(DOMUtil.java:156) at org.apache.solr.util.DOMUtil.nodesToNamedList(DOMUtil.java:117) at org.apache.solr.util.DOMUtil.childNodesToNamedList(DOMUtil.java:107) at org.apache.solr.util.DOMUtil.addToNamedList(DOMUtil.java:156) at org.apache.solr.util.DOMUtil.nodesToNamedList(DOMUtil.java:117) at org.apache.solr.util.DOMUtil.childNodesToNamedList(DOMUtil.java:107) at org.apache.solr.util.DOMUtil.addToNamedList(DOMUtil.java:156) at org.apache.solr.util.DOMUtil.nodesToNamedList(DOMUtil.java:117) at org.apache.solr.util.DOMUtil.childNodesToNamedList(DOMUtil.java:107) at org.apache.solr.util.DOMUtil.addToNamedList(DOMUtil.java:156)

Note: the Solr version we are using - 8.11.1

pareekdevanshu commented 6 months ago

also, the same issue with Solr operator version 0.7.1. We tried upgrading the operator to version 0.8.0 but then the Prometheus exporter failed to start with the below error:`

org.apache.solr.prometheus.exporter.SolrExporter; Start server failed: org.apache.solr.core.SolrResourceNotFoundException: Can't find resource 'solr-exporter-config.xml' in classpath or '/opt/solr/conf'

HoustonPutman commented 6 months ago

Are you using the official Solr docker image?

pareekdevanshu commented 6 months ago

we are using official Solr docker image as base image, copying our own log4j2.xml into it and using it.

HoustonPutman commented 6 months ago

Can you output the kubectl describe for your SolrPrometheusExporter and the pod that it creates?

pareekdevanshu commented 6 months ago

we found the problem, SolrPrometheusExporter was using older version of Solr image, we upgraded it to Solr version 8.11 and it fixed the problem.