noamt / elasticsearch-grails-plugin

The revived ElasticSearch grails plugin
Based on Graeme Rocher initial stub. Note that it is still in early stage.
Other
63 stars 83 forks source link

Indexing issue with ElasticSearch 2.4.0 and Grails 3.3.2 #216

Closed mcam11 closed 5 years ago

mcam11 commented 6 years ago

I am new to ElasticSearch (was using Searchable plugin before with Grails 2.4.4) and am getting the following error when starting up my Grails 3.3.2 application. Note: previously I submitted issue #214 which was fixed, but am now getting a different error.

2018-02-02 12:47:14.963 ERROR --- [           main] g.p.e.index.IndexRequestQueue            : Error 
Indexing class com.xxx.server.Company_$$_javassist_32 (index: com.xxx.server_write , type: company) of id 1

grails.plugins.elasticsearch.exception.IndexException: Failed to marshall domain instance [Yyy]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.springsource.loaded.ri.ReflectiveInterceptor.jlrConstructorNewInstance(ReflectiveInterceptor.java:1076)
    at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
    at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
    at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:255)
    at grails.plugins.elasticsearch.index.IndexRequestQueue.toJSON(IndexRequestQueue.groovy:108)
    at sun.reflect.GeneratedMethodAccessor256.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1427)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
    at grails.plugins.elasticsearch.index.IndexRequestQueue$_executeRequests_closure1.doCall(IndexRequestQueue.groovy:149)
    at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1427)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at groovy.lang.Closure.call(Closure.java:414)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForMapEntry(DefaultGroovyMethods.java:5276)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2117)
    at org.codehaus.groovy.runtime.dgm$164.invoke(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
    at grails.plugins.elasticsearch.index.IndexRequestQueue.executeRequests(IndexRequestQueue.groovy:143)
    at grails.plugins.elasticsearch.index.IndexRequestQueue$executeRequests$2.call(Unknown Source)
    at grails.plugins.elasticsearch.ElasticSearchService$_doBulkRequest_closure2.doCall(ElasticSearchService.groovy:291)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1427)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at groovy.lang.Closure.call(Closure.java:414)
    at groovy.lang.Closure.call(Closure.java:430)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2040)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2025)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2066)
    at org.codehaus.groovy.runtime.dgm$163.invoke(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
    at grails.plugins.elasticsearch.ElasticSearchService.doBulkRequest(ElasticSearchService.groovy:257)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1427)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
    at grails.plugins.elasticsearch.ElasticSearchService.index(ElasticSearchService.groovy:164)
    at grails.plugins.elasticsearch.ElasticSearchBootStrapHelper.bulkIndexOnStartup(ElasticSearchBootStrapHelper.groovy:34)
    at grails.plugins.elasticsearch.ElasticSearchBootStrapHelper$bulkIndexOnStartup.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:80)
    at ElasticsearchBootStrap$_closure1.doCall(ElasticsearchBootStrap.groovy:8)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1427)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1087)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at groovy.lang.Closure.call(Closure.java:414)
    at groovy.lang.Closure.call(Closure.java:408)
    at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:541)
    at grails.util.Environment.executeForEnvironment(Environment.java:534)
    at grails.util.Environment.executeForCurrentEnvironment(Environment.java:510)
    at org.grails.web.servlet.boostrap.DefaultGrailsBootstrapClass.callInit(DefaultGrailsBootstrapClass.java:74)
    at org.grails.web.servlet.context.GrailsConfigUtils.executeGrailsBootstraps(GrailsConfigUtils.java:65)
    at org.grails.plugins.web.servlet.context.BootStrapClassRunner.onStartup(BootStrapClassRunner.groovy:53)
    at grails.boot.config.GrailsApplicationPostProcessor.onApplicationEvent(GrailsApplicationPostProcessor.groovy:261)
    at grails.boot.config.GrailsApplicationPostProcessor.onApplicationEvent(GrailsApplicationPostProcessor.groovy)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:883)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
    at grails.boot.GrailsApp.run(GrailsApp.groovy:83)
    at grails.boot.GrailsApp.run(GrailsApp.groovy:387)
    at grails.boot.GrailsApp.run(GrailsApp.groovy:374)
    at grails.boot.GrailsApp$run.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
    at gzserver.Application.main(Application.groovy:8)
Caused by: groovy.lang.GroovyRuntimeException: Ambiguous method overloading for method 
grails.plugins.elasticsearch.ElasticSearchContextHolder#getMappingContext.
Cannot resolve which method to invoke for [null] due to overlapping prototypes between:
    [class grails.plugins.elasticsearch.mapping.DomainEntity]
    [class java.lang.String]
    at groovy.lang.MetaClassImpl.chooseMostSpecificParams(MetaClassImpl.java:3263)
    at groovy.lang.MetaClassImpl.chooseMethodInternal(MetaClassImpl.java:3216)
    at groovy.lang.MetaClassImpl.chooseMethod(MetaClassImpl.java:3159)
    at groovy.lang.MetaClassImpl.getMethodWithCachingInternal(MetaClassImpl.java:1331)
    at groovy.lang.MetaClassImpl.createPogoCallSite(MetaClassImpl.java:3426)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.createPogoSite(CallSiteArray.java:150)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(CallSiteArray.java:164)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at grails.plugins.elasticsearch.ElasticSearchContextHolder$getMappingContext.call(Unknown Source)
    at grails.plugins.elasticsearch.conversion.JSONDomainFactory.buildJSON(JSONDomainFactory.groovy:145)
    at grails.plugins.elasticsearch.conversion.JSONDomainFactory$buildJSON.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at grails.plugins.elasticsearch.conversion.JSONDomainFactory$buildJSON.call(Unknown Source)
    at grails.plugins.elasticsearch.index.IndexRequestQueue.toJSON(IndexRequestQueue.groovy:106)
    ... 109 common frames omitted

Here is my build.gradle:

def esVersion = '5.4.1'
ext['elasticsearch.version'] = esVersion

ext {
    elasticsearchVersion = esVersion
}

dependencies {
...
    compile 'org.grails.plugins:elasticsearch:2.4.0'
    runtime 'org.elasticsearch.plugin:mapper-attachments:2.4.6'
...
}

Here is my configuration code from application.groovy:

elasticSearch.datastoreImpl = 'hibernateDatastore'
elasticSearch.client.mode = 'local'
elasticSearch.migration.strategy = 'delete'
elasticSearch.disableAutoIndex = false
elasticSearch.bulkIndexOnStartup = true
puneetbehl commented 6 years ago

Please upload a sample application replicating the problem. Also, look into the Demo App.

mcam11 commented 6 years ago

I was able to reproduce with the following two domains. If I remove the 3 mentions of 'company' from within the Employee domain, then all works fine.

class Company {
    static searchable = {
        only = ['companyName', 'state', 'country']
    }

    static hasMany = [employees: Employee]

    String companyName
    String address
    String address2
    String city
    String state
    String country
    Date created = new Date()

    static constraints = {
        companyName(blank: false, size: 1..255)
        address(blank: false, size: 1..255)
        address2(nullable: true, maxSize: 255)
        city(blank: false, size: 1..255)
        state(nullable: true)
        country(nullable: true)
    }

    static mapping = {
    table 'COMPANY'
    }
}

and

class Employee {

    static searchable = {
        only = ['name', 'email', 'position']
    }

    static belongsTo = [Company]

    String name
    String email
    String position
    Double salary
    Date created = new Date()
    Company company

    static constraints = {
        email email: true
    company(nullable: true)
    }

    static mapping = {
        table 'EMPLOYEE'
    }
}

I started with a new Grails project. Added these two domains + crud.

Added the following to my build.gradle (the runtime mapper-attachments was necessary) and I used a MySQL database to store data. The error happens when you start up the application with stored data that then needs to be indexed on app startup.

def esVersion = '5.4.1'
ext['elasticsearch.version'] = esVersion

dependencies {
    compile 'org.grails.plugins:elasticsearch:2.4.0'             
    runtime "org.elasticsearch.plugin:mapper-attachments:2.4.6"   
    runtime "mysql:mysql-connector-java:5.1.37"     
}            

Created an application.groovy with these contents (to go along with existing application.yml):

elasticSearch.datastoreImpl = 'hibernateDatastore'
elasticSearch.client.mode = 'local'
elasticSearch.migration.strategy = 'delete'
elasticSearch.disableAutoIndex = false
elasticSearch.bulkIndexOnStartup = true

Modified application.yml to use a MySQL database for prod:

dataSource:
    pooled: true
    jmxExport: true
    driverClassName: com.mysql.jdbc.Driver
    username: sa
    password: ''

environments:
    development:
        dataSource:
            dbCreate: create-drop
            url: jdbc:h2:mem:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
    test:
        dataSource:
            dbCreate: update
            url: jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
    production:
        dataSource:
            dbCreate: update
            url: jdbc:mysql://localhost:3306/mydb
            username: 'mydbuser'
            password: 'mydbpassword'
            properties:
                jmxEnabled: true
                initialSize: 5
                maxActive: 50
                minIdle: 5
                maxIdle: 25
                maxWait: 10000
                maxAge: 600000
                timeBetweenEvictionRunsMillis: 5000
                minEvictableIdleTimeMillis: 60000
                validationQuery: SELECT 1
                validationQueryTimeout: 3
                validationInterval: 15000
                testOnBorrow: true
                testWhileIdle: true
                testOnReturn: true

I created an empty MySQL database with db permissions, launched app (grails prod run-app), and created a company and employee. Shutdown app. Relaunched and got the error.

I'm not sure how to upload a sample application.

mcam11 commented 6 years ago

Also meant to say that I can search one domain or the other, but not both. So if I comment out "static searchable ..." on Company, it works for Employee, and vice versa.

mcam11 commented 6 years ago

I think the issue is that my Employee domain contains an instance of my Company domain and that instance can be null.

If I get rid of the following constraint in my Employee domain, it works; I am able to successfully launch the application ten out of ten times with no indexing issues of my data.

static constraints = {
    ...
    company(nullable: true)
}

Also, when it doesn't work (the original domains above) it is intermittent. If I run the app and create 3 companies and 3 employees, then exit, when I restart the app ten times it fails about five times with the original error shown above.

puneetbehl commented 6 years ago

From your latest comment, it seems like the issue is not with Elasticsearch plugin.

Also, you can follow Grails Guide as a starting point for your project.

To answer your questions "I'm not sure how to upload a sample application." You can create a sample application with minimum code, demonstrating the problem and then push it to a new GitHub repository so that I can test it by myself.

jharlan-nextcentury commented 6 years ago

I think I ran across this same issue, had to do with Hibernate proxies not being unwrapped in IndexRequestQueue and JSONDomainFactory.

I'll try to submit a pull request with the fix as soon as I can.

alanlit commented 6 years ago

I think I ran across this same issue, had to do with Hibernate proxies not being unwrapped in IndexRequestQueue and JSONDomainFactory.

I'll try to submit a pull request with the fix as soon as I can.

I was having exactly the same issue (Grails 3.3.2, ES Plugin 2.4.1 and Hibernate) and this fix resolved it.