search

AtlasOfLivingAustralia / biocache-store

Occurrence processing, indexing and batch processing
Other
7 stars 24 forks source link

occurrenceYear appears to be multivalued in solr but not in hardcoded scala object model #310

Closed ansell closed 5 years ago

ansell commented 5 years ago

Solr indexes generated with biocache-store-2.4.2 appear to be able to create multivalued fields for occurrenceYear. It isn't clear why occurrenceYear would suddenly be converted to multivalued, but given the indexing is based on code and not configuration driven, it may have always been that way and was being optimised down to a scalar in the past before a fix to date parsing was made.

This creates the following issue when attempting to deserialise a results document from Solr using the hardcoded Scala object model:

Because of this issue, we had to manually rollback to a previous index. The index causing the issue has been hardlinked from biocache-ws-test to debug the issue, and can be replicated using:

https://biocache-ws-test.ala.org.au/ws/occurrences/search?taxa=wombat

The error message from that resource:

{
  "pageSize": 10,
  "startIndex": 0,
  "totalRecords": 0,
  "dir": "asc",
  "status": "ERROR",
  "errorMessage": "Could not instantiate object of class au.org.ala.biocache.dto.OccurrenceIndex"
}

matches the underlying biocache-service log message:

==> /var/log/tomcat7/biocache-service.log <==
2019-01-18 07:30:47,089 [http-bio-8080-exec-4] ERROR au.org.ala.biocache.dao.SearchDAOImpl  (SearchDAOImpl.java:689) - Error executing query with requestParams: q=*:*&start=0&pageSize=10&sort=score&dir=asc&qc=&formattedQuery=*:*&facet=true
org.apache.solr.client.solrj.beans.BindingException: Could not instantiate object of class au.org.ala.biocache.dto.OccurrenceIndex
    at org.apache.solr.client.solrj.beans.DocumentObjectBinder.getBean(DocumentObjectBinder.java:71)
    at org.apache.solr.client.solrj.beans.DocumentObjectBinder.getBeans(DocumentObjectBinder.java:50)
    at org.apache.solr.client.solrj.response.QueryResponse.getBeans(QueryResponse.java:618)
    at au.org.ala.biocache.dao.SearchDAOImpl.processSolrResponse(SearchDAOImpl.java:2754)
    at au.org.ala.biocache.dao.SearchDAOImpl.findByFulltextSpatialQuery(SearchDAOImpl.java:678)
    at au.org.ala.biocache.dao.SearchDAOImpl.findByFulltextSpatialQuery(SearchDAOImpl.java:653)
    at sun.reflect.GeneratedMethodAccessor619.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
    at com.sun.proxy.$Proxy44.findByFulltextSpatialQuery(Unknown Source)
    at au.org.ala.biocache.web.OccurrenceController.occurrenceSearch(OccurrenceController.java:819)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at au.org.ala.biocache.web.SimpleCORSFilter.doFilter(SimpleCORSFilter.java:33)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at au.org.ala.biocache.web.LoggerConfigFilter.doFilter(LoggerConfigFilter.java:26)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.beans.BindingException: Exception while setting value : [Sat Jan 01 20:00:00 AEST 1910, Sat Jan 01 10:00:00 AEST 1910] on java.util.Date au.org.ala.biocache.dto.OccurrenceIndex.occurrenceYear
    at org.apache.solr.client.solrj.beans.DocumentObjectBinder$DocField.set(DocumentObjectBinder.java:455)
    at org.apache.solr.client.solrj.beans.DocumentObjectBinder$DocField.inject(DocumentObjectBinder.java:438)
    at org.apache.solr.client.solrj.beans.DocumentObjectBinder.getBean(DocumentObjectBinder.java:67)
    ... 62 more
Caused by: java.lang.IllegalArgumentException: Can not set java.util.Date field au.org.ala.biocache.dto.OccurrenceIndex.occurrenceYear to java.util.ArrayList
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
    at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
    at sun.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81)
    at java.lang.reflect.Field.set(Field.java:764)
    at org.apache.solr.client.solrj.beans.DocumentObjectBinder$DocField.set(DocumentObjectBinder.java:449)
    ... 64 more

==> /var/log/nginx/access.log <==
[REDACTED] - - [18/Jan/2019:07:30:47 +1100] "GET /ws/occurrences/search?taxa=wombat HTTP/1.1" 200 164 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:64.0) Gecko/20100101 Firefox/64.0" "[REDACTED]" request_time=0.592 upstream_response_time=0.592 upstream_connect_time=0.000 upstream_header_time=0.592 upstream_cache_status=-
ansell commented 5 years ago

occurrence_year does not appear to be declared as multivalued in the solr schema of either the latest or the second latest solr collections:

http://localhost:8983/solr/biocache-2019-01-17-10-10/schema/fields?wt=json
{"name":"occurrence_year","type":"tdate","multiValued":false,"indexed":true,"stored":true}
vjrj commented 5 years ago

After upgrade our biocache-store to 2.4.2 we suffered this bug. So we did a re-index using biocache-store 2.4.3 with some similar error in biocache-service: 

Caused by: java.lang.IllegalArgumentException: Can not set java.lang.Integer field au.org.ala.biocache.dto.OccurrenceIndex.year to java.lang.String.
(...)
Caused by: org.apache.solr.client.solrj.beans.BindingException: Exception while setting value : 1987 on java.lang.Integer au.org.ala.biocache.dto.OccurrenceIndex.year
        at org.apache.solr.client.solrj.beans.DocumentObjectBinder$DocField.set(DocumentObjectBinder.java:455)
        at org.apache.solr.client.solrj.beans.DocumentObjectBinder$DocField.inject(DocumentObjectBinder.java:438)
        at org.apache.solr.client.solrj.beans.DocumentObjectBinder.getBean(DocumentObjectBinder.java:67)

So now I'm trying to re-index with biocache-store 2.4.1.

Meanwhile I have tried several biocache-service versions 2.1.{11,13,14,15,16-SNAPSHOT} with same results. Our sorl (6.6.5) & cassandra (3.11.3) are also up-to-date.

Any recommendation on compatible versions? By the way, I'm trying to update this: https://github.com/AtlasOfLivingAustralia/documentation/wiki/Components-versioning

PS: crossposting from slack channel