search

AtlasOfLivingAustralia / biocache-store

Occurrence processing, indexing and batch processing
Other
7 stars 24 forks source link

docValues error on sensitive_latitude/sensitive_longitude #325

Closed ansell closed 5 years ago

ansell commented 5 years ago

The following error is occurring during biocache queries after reindexing using the docValues code added in https://github.com/AtlasOfLivingAustralia/biocache-store/commit/f52069d6ec039c3898fdd81dd71e5c7463fe5eec :

aws-sc5b logs:

2019-02-13 02:33:51.483 ERROR (qtp120960120-1269) [c:biocache-2019-02-13-06-40 s:shard5 r:core_node5 x:biocache-2019-02-13-06-40_shard5_replica0] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Exception during facet.field: sensitive_longitude
    at org.apache.solr.request.SimpleFacets.lambda$getFacetFieldCounts$0(SimpleFacets.java:809)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.solr.request.SimpleFacets$3.execute(SimpleFacets.java:742)
    at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:818)
    at org.apache.solr.handler.component.FacetComponent.getFacetCounts(FacetComponent.java:330)
    at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:274)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
    at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
    at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:534)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: unexpected docvalues type NONE for field 'sensitive_longitude' (expected=NUMERIC). Re-index with correct docvalues type.
    at org.apache.lucene.index.DocValues.checkField(DocValues.java:212)
    at org.apache.lucene.index.DocValues.getNumeric(DocValues.java:227)
    at org.apache.solr.request.NumericFacets.getCountsSingleValue(NumericFacets.java:218)
    at org.apache.solr.request.NumericFacets.getCounts(NumericFacets.java:169)
    at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:499)
    at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:405)
    at org.apache.solr.request.SimpleFacets.lambda$getFacetFieldCounts$0(SimpleFacets.java:803)
    ... 39 more

prod-bws-b5-1 logs:


==> /var/log/tomcat7/biocache-service.log <==
2019-02-13 13:32:52,264 [http-bio-127.0.0.1-8080-exec-22] INFO au.org.ala.biocache.web.ExploreController  (ExploreController.java:298) - Points search for point-0.001 - found: 966
2019-02-13 13:32:52,677 [http-bio-127.0.0.1-8080-exec-57] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient  (CloudSolrClient.java:1167) - Request to collection biocache failed due to (500) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc5b.ala:8983/solr/biocache: Error from server at http://aws-sc5b.ala:8983/solr/biocache-2019-02-13-06-40_shard5_replica0: Exception during facet.field: sensitive_latitude, retry? 0
2019-02-13 13:32:52,677 [http-bio-127.0.0.1-8080-exec-57] ERROR au.org.ala.biocache.dao.SearchDAOImpl  (SearchDAOImpl.java:689) - Error executing query with requestParams: q=sensitive:*&start=0&pageSize=10&sort=score&dir=asc&qc=&facets=sensitive_latitude&formattedQuery=sensitive:*&facet=true
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://aws-sc2b.ala:8983/solr/biocache, http://aws-sc3b.ala:8983/solr/biocache, http://aws-sc4b.ala:8983/solr/biocache, http://aws-sc5b.ala:8983/solr/biocache, http://aws-sc7b.ala:8983/solr/biocache, http://aws-sc8b.ala:8983/solr/biocache, http://aws-sc6b.ala:8983/solr/biocache, http://aws-sc1b.ala:8983/solr/biocache]
    at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:426)
    at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383)
    at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
    at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
    at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
    at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:974)
    at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:990)
    at au.org.ala.biocache.dao.SearchDAOImpl.query(SearchDAOImpl.java:3959)
    at au.org.ala.biocache.dao.SearchDAOImpl.runSolrQuery(SearchDAOImpl.java:2683)
    at au.org.ala.biocache.dao.SearchDAOImpl.findByFulltextSpatialQuery(SearchDAOImpl.java:675)
    at au.org.ala.biocache.dao.SearchDAOImpl.findByFulltextSpatialQuery(SearchDAOImpl.java:653)
    at sun.reflect.GeneratedMethodAccessor240.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
    at com.sun.proxy.$Proxy44.findByFulltextSpatialQuery(Unknown Source)
    at au.org.ala.biocache.web.OccurrenceController.occurrenceSearch(OccurrenceController.java:819)
    at sun.reflect.GeneratedMethodAccessor275.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at au.org.ala.biocache.web.SimpleCORSFilter.doFilter(SimpleCORSFilter.java:33)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at au.org.ala.biocache.web.LoggerConfigFilter.doFilter(LoggerConfigFilter.java:26)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc5b.ala:8983/solr/biocache: Error from server at http://aws-sc5b.ala:8983/solr/biocache-2019-02-13-06-40_shard5_replica0: Exception during facet.field: sensitive_latitude
    at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:612)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
    at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
    at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388)
    ... 66 more

==> /var/log/nginx/access.log <==
[REDACTED] - - [13/Feb/2019:13:31:25 +1100] "GET /ws/occurrences/search?q=sensitive:*&facet=true&facets=sensitive_longitude HTTP/1.1" 200 211 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36" "[REDACTED]" request_time=0.048 upstream_response_time=0.048 upstream_connect_time=0.000 upstream_header_time=0.048 upstream_cache_status=-
adam-collins commented 5 years ago

I mistakenly ran a query on prod instead of my local environment to produce this error. I am working through the biocache-service implications when using docvalues.

Regardless of the error message reported the error is due to an attempt to facet on a SOLR field that is indexed=false and docValues=false.