Solandra: Issue with shards

[compfix]

We've been able to setup our index in the Solandra edition (Lucandra and Cassandra in the same JVM) but once the index shards our queries no longer work. In the SolandraComponent I see the following line: String shard = addr.getHostAddress() + ":8983/solr/" + indexName + "~" + i; which references a solr instance in jetty, however that no longer appears to be part of the build.

Is this a mistake or is it expected that a solr instance is installed along side the solandra webapp ?

Thanks in advance.

[compfix]

I was able to get past the original issue by changing the String shard = addr.getHostAddress() + ":8983/solr/" + indexName + "~" + i; to String shard = addr.getHostAddress() + ":8983/solandra/" + indexName + "~" + i;

However we are now getting a null pointer exception:

15:18:58,292 ERROR SolrCore:139 - java.lang.NullPointerException at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:462) at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:298) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:290) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at solandra.SolandraDispatchFilter.execute(SolandraDispatchFilter.java:169) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at solandra.SolandraDispatchFilter.doFilter(SolandraDispatchFilter.java:133) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:722) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:404) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:828) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

when we run a query.

[tjake]

Did you update recently? I may have broken this while renaming things, I'll look at this tomorrow on the plane

[tjake]

Well I added your fix for now. Have your upgraded all nodes to the latest?

[compfix]

I'm planning on pulling your latest tomorrow. I think my issue right now is that some of the documents come back with null column families which cause the returned document to be null. Perhaps an issue with cassandra? I'll let you know if my results are better after I pull the latest.

Thanks,

Cliff

[compfix]

Jake,

It appears as those Solandra is based on the assumption that there will be multiple nodes in the cassandra cluster that will store the "shards". Is this the case? What if we only have one node? Can we prevent Solandra from thisnking there are multiple shards once our doc count goes beyond the 2^17 limit on maxDocsPerShard ?

Thanks,

Cliff

[tjake]

Yes, just set SHARDS_AT_ONCE="1" in start-solandra.sh

[compfix]

I did see that setting and it has been set that way. Unfortunately we still experience a problem. Based on the log information, it seems to pull a result from the database, however it fails to merge the results to present to the user because it reports "Missing document in multiget_slice for" ... I'm not sure if this is because the doc isn't in the database or because it thinks it actually resides in a different shard the cassandra isn't aware of.

I'm able to reproduce this fairly easily but haven't made much headway in determining root cause. Any guidance on this would be appreciated.

Thanks,

Cliff

[tjake]

Can you post the log?

[compfix]

Here's the excerpt from the log when I query: (There are two Solr cores one called config and one called log) The one called log is the one with issues.

16:17:57,306 INFO SolandraCoreContainer:72 - Loading Solandra core: log~0 16:17:57,306 INFO SolandraCoreContainer:72 - Loading Solandra core: log~1 16:17:57,338 INFO SolandraCoreContainer:72 - Loading Solandra core: config 16:17:57,340 INFO UpdateRequestProcessor:171 - {add=[3e7d197b-2582-4847-a6fa-6bea1b13aecf:current]} 0 2 16:17:57,360 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b88 16:17:57,342 INFO SolandraCoreContainer:72 - Loading Solandra core: log 16:17:57,362 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b89 16:17:57,362 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b82 16:17:57,362 INFO SolrCore:1324 - [config] webapp=/solandra path=/update params={} status=0 QTime=2 16:17:57,363 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b83 16:17:57,364 INFO UpdateRequestProcessor:171 - {add=[bc28ab8a-c92d-40b5-b164-d6cd028b92f1]} 0 2 16:17:57,364 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b80 16:17:57,364 INFO SolrCore:1324 - [log] webapp=/solandra path=/update params={} status=0 QTime=2 16:17:57,365 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b81 16:17:57,365 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b86 16:17:57,365 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b87 16:17:57,365 WARN IndexReader:254 - Missing document in multiget_slice for: 8441519560042444830915831745756986754ï¿¿5b84 16:17:57,365 WARN IndexReader:254 - Missing document in multigetslice for: 8441519560042444830915831745756986754ï¿¿5b85 16:17:57,366 INFO SolrCore:1324 - [log] webapp=/solandra path=/select params={fl=id,score&start=0&q=uuid:&isShard=true&wt=javabin&fsv=true&rows=10&version=1} hits=547 status=0 QTime=60 16:17:57,389 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿189 16:17:57,393 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿188 16:17:57,393 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿183 16:17:57,393 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿182 16:17:57,393 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿181 16:17:57,394 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿180 16:17:57,394 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿187 16:17:57,394 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿186 16:17:57,394 WARN IndexReader:254 - Missing document in multiget_slice for: 89333738597305042842099228572829323411ï¿¿185 16:17:57,394 WARN IndexReader:254 - Missing document in multigetslice for: 89333738597305042842099228572829323411ï¿¿184 16:17:57,394 INFO SolrCore:1324 - [log] webapp=/solandra path=/select params={fl=id,score&start=0&q=uuid:&isShard=true&wt=javabin&fsv=true&rows=10&version=1} hits=33536 status=0 QTime=88 16:17:57,396 INFO SolandraCoreContainer:72 - Loading Solandra core: config 16:17:57,401 INFO SolandraCoreContainer:72 - Loading Solandra core: log 16:17:57,402 INFO UpdateRequestProcessor:171 - {add=[29595742-3036-4dab-bd38-c7fecec28da0:current]} 0 5 16:17:57,402 INFO SolrCore:1324 - [config] webapp=/solandra path=/update params={} status=0 QTime=5 16:17:57,408 INFO UpdateRequestProcessor:171 - {add=[a0c7f456-5a98-4ed7-bbac-3ef8cbc4f146]} 0 6 16:17:57,408 INFO SolrCore:1324 - [log] webapp=/solandra path=/update params={} status=0 QTime=6 16:17:57,408 ERROR SolrCore:139 - java.lang.NullPointerException at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:462) at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:298) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:290) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at solandra.SolandraDispatchFilter.execute(SolandraDispatchFilter.java:169) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at solandra.SolandraDispatchFilter.doFilter(SolandraDispatchFilter.java:133) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:722) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:404) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:828) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

16:17:57,408 INFO SolrCore:1324 - [log] webapp=/solandra path=/select params={q=uuid:*} status=500 QTime=568 16:17:57,409 ERROR SolrDispatchFilter:139 - java.lang.NullPointerException at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:462) at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:298) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:290) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at solandra.SolandraDispatchFilter.execute(SolandraDispatchFilter.java:169) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at solandra.SolandraDispatchFilter.doFilter(SolandraDispatchFilter.java:133) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:722) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:404) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:828) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

16:17:57,438 INFO SolandraCoreContainer:72 - Loading Solandra core: config

You can get the full log here: https://docs.google.com/leaf?id=0BxyBcfQyaiLCMzM1NTA1NmMtZjc3My00NGY1LWJlZWMtMDFkOGQ4YzU2MDk1&hl=en&authkey=CNriscMO

[tjake]

Hmm, have you been deleting documents? could be a bug in the delete logic leaving behind old references

[compfix]

No. Everything is an add at this point. We aren't deleting anything.

[tjake]

What changes have you made to the code and/or settings?

BTW, you can chat me directly on irc freenode #solandra

[tjake]

Could you also post the schema.xml you are using?

[tjake]

This ended up being the in schema.xml wasn't marked stored="true"