Failed bulk item: UnavailableShardsException

[berinle]

I am not sure exactly what the problem is here, but on a vanilla project, once ES plugin is added, my tests seem to hang when I run test-app command.

Even in cases where it manages to run through to the very end, I start to see these errors...

| Error 2012-02-09 23:09:08,059 [elasticsearch[cached]-pool-8-thread-6] ERROR index.IndexRequestQueue - Failed bulk item: UnavailableShardsException[[com.mycompany.app1][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@10f00821] | Error 2012-02-09 23:09:08,060 [elasticsearch[cached]-pool-8-thread-6] ERROR index.IndexRequestQueue - Failed bulk item: UnavailableShardsException[[com.mycompany.app1][1] [1] shardIt, [0] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@3b0240db]

A timeout. I also notice there is be a lock that is not released on the "data" directory ES is keeping its indices in.

I have even attempted to change the configurations to in memory only (which is the default for test environment anyway), but that didn't fix the problem.

elasticSearch { bulkIndexOnStartup = false
index.store.type = 'memory' }

Please advice.

PS: sample project @ https://github.com/berinle/grails-search-app

Steps to reproduce: Clone and then run test-app

[berinle]

Turning on logging for ES plugin and attempting to run test-app on the same project seems to be taking forever. It took sometime to recognize an index already existed, after that, no idea what it is doing now. It's been going for almost 10 mins now or this very simple vanilla application. I pushed the change so you can observe yourself.

berinle:grails-search-app/ (master✗) $ grails test-app [10:49:32] | Compiling 72 source files Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. | Completed 10 unit tests, 0 failed in 2944ms | Packaging Grails application..... 2012-02-14 10:49:57,937 [main] DEBUG elasticsearch.ClientNodeFactoryBean - Local ElasticSearch client with store type of memory configured. 2012-02-14 10:49:59,239 [main] DEBUG mapping.SearchableClassMappingConfigurator - Retrieved index settings 2012-02-14 10:49:59,240 [main] DEBUG mapping.SearchableClassMappingConfigurator - Installing mappings... 2012-02-14 10:52:14,936 [main] DEBUG mapping.SearchableClassMappingConfigurator - Index com.mycompany.app1 already exists, skip index creation. 2012-02-14 10:52:14,936 [main] DEBUG mapping.SearchableClassMappingConfigurator - [foo] => {foo={properties={name={type=string, include_in_all=true, term_vector=with_positions_offsets}}}}

No errors yet on the console.

[mstein]

Just pulled your app, and it started properly.

After the mapping process, the plugin wait for the ES instance to get into the "YELLOW" status at least. If it never reach that status, it may means that the ES instance encountered some issue when trying to start the master node.

Perhaps you still had that "lock" issue you described earlier ?

[berinle]

I'll do a clean and recheck. If the index mode is set to 'memory', where and why will it be holding a lock? Sorry, I am new to ES, but familiar with search in general.

[berinle]

I did pull the app under a different name and it worked fine -- the first time!

A subsequent run (of grails test-app) is now stuck in the same exact position as above. So something is definitely up somewhere.

After pulling the app, I did 'grails test-app', which worked fine.

After that completed, I tried to run the test suite again and it is now stuck again.

Since the test cases all stored indexes in memory, I don't know why a held lock will exist or not be released.

Please advice

[berinle]

mstein - just an update. the test case does eventually finish after 5 mins. I just timed it. Is that the norm? It seems a rather long time to run test cases in this simple app and considering the first time I ran it, it took about 1 min or so.

Is this something that is configurable via the plugin? i.e. time to wait for ES to turn green, or the checks for if an index already exists, or remapping the indexes, etc

[mstein]

5 mins is definitely too long. For the "in-memory" storage in ES : even with that setting, ES will still need the file system for a few thing, like the transaction logs, so it will still create a data folder, but it shouldn't create any file to store the indices.

[berinle]

ok, thx for explaining the files needed even with in memory setting.

I also think 5 mins too long. Any ideas how we can cross this hurdle? Is this an ES issue or the way the plugin is interacting with it?