search

AtlasOfLivingAustralia / biocache-store

Occurrence processing, indexing and batch processing
Other
7 stars 24 forks source link

Indexing required by Flag an issue in biocache-hubs/biocache-service has a data race with resampling and reindexing #317

Closed ansell closed 5 years ago

ansell commented 5 years ago

The Flag an issue feature in biocache-hubs has been reported by a number of users as not working recently, and has been reproduced, as described in https://github.com/AtlasOfLivingAustralia/biocache-hubs/issues/140 That issue is closed because it is currently working, and it is not a bug in biocache-hubs.

The reason the failure occurs is related to adding new layers and has the following steps that cause periods where annotations are not successful due to the solr schema not matching the cassandra record contents on occ.loc and occ.occ.cl/occ.occ.el:

This is fairly urgent to fix or mitigate somehow, as we are planning to load many new layers in the near future.

ansell commented 5 years ago

An example from a few seconds ago of an error message in biocache-service (on prod-bws-b5-2) related to this issue:

==> /var/log/tomcat7/biocache-service.log <==
2019-01-25 09:42:59,338 [http-bio-8080-exec-2172] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient  (CloudSolrClient.java:1167) - Request to collection biocache failed due to (400) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc4b.ala:8983/solr/biocache-2019-01-23-06-49_shard4_replica0: ERROR: [doc=1b4e34d2-4285-4b3f-ab93-37ce2a4fedca] unknown field 'cl12021', retry? 0
2019-01-25 09:42:59,339 [http-bio-8080-exec-2172] ERROR au.org.ala.biocache.web.AssertionController  (AssertionController.java:209) - Error from server at http://aws-sc4b.ala:8983/solr/biocache-2019-01-23-06-49_shard4_replica0: ERROR: [doc=1b4e34d2-4285-4b3f-ab93-37ce2a4fedca] unknown field 'cl12021'
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://aws-sc4b.ala:8983/solr/biocache-2019-01-23-06-49_shard4_replica0: ERROR: [doc=1b4e34d2-4285-4b3f-ab93-37ce2a4fedca] unknown field 'cl12021'
    at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:819)
    at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1263)
    at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
    at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
    at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
    at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
    at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:190)
    at au.org.ala.biocache.index.SolrIndexDAO.indexFromMap(SolrIndexDAO.scala:826)
    at au.org.ala.biocache.dao.OccurrenceDAOImpl.reIndex(OccurrenceDAOImpl.scala:1242)
    at au.org.ala.biocache.Store$.addUserAssertion(Store.scala:346)
    at au.org.ala.biocache.Store.addUserAssertion(Store.scala)
    at au.org.ala.biocache.web.AssertionController.addAssertion(AssertionController.java:203)
    at au.org.ala.biocache.web.AssertionController.addAssertionWithParams(AssertionController.java:119)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at au.org.ala.biocache.web.SimpleCORSFilter.doFilter(SimpleCORSFilter.java:33)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at au.org.ala.biocache.web.LoggerConfigFilter.doFilter(LoggerConfigFilter.java:26)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc4b.ala:8983/solr/biocache-2019-01-23-06-49_shard4_replica0: ERROR: [doc=1b4e34d2-4285-4b3f-ab93-37ce2a4fedca] unknown field 'cl12021'
    at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:612)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
    at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
    at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388)
    at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:796)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more

==> /var/log/nginx/access.log <==
[REDACTED] - - [25/Jan/2019:09:42:59 +1100] "POST /ws/occurrences/assertions/add HTTP/1.1" 400 0 "-" "-" "[REDACTED]" request_time=0.066 upstream_response_time=0.066 upstream_connect_time=0.000 upstream_header_time=0.066 upstream_cache_status=-
ansell commented 5 years ago

Helpdesk tickets referencing this issue include: