NPE in BytesRefOrdValComparator

[ghost]

This issue relates to issue #3189 and I'm experiencing it with v1.0.0.Beta, but I guess that applies to 0.90 branch as well.

While sorting on some trivial fields I get the following error message:

{
   "error": "SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed; shardFailures {[UIomKhmqSzumD0uzjmRbfA][brisa][0]: QueryPhaseExecutionException[[brisa][0]: query[filtered(ConstantScore(+NotFilter(cache(discarded:T))))->cache(_type:patients)],from[0],size[20],sort[<custom:\"lastname\": org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource@1d8f4c01>]: Query Failed [Failed to execute main query]]; nested: NullPointerException; }]",
   "status": 500
}

lastname mapping is as follows:

"lastname": {
  "type": "string",
  "index": "not_analyzed",
  "omit_norms": true,
  "index_options": "docs"
}

Fairly standard IMO.

Now the funny thing is that I have another name mapping which is exactly the same and everything works just fine. From a data point of view, I cannot see a difference between one field and the other (upper case strings).

The NPE happens in BytesRefOrdValComparator at line 388 because MultiDocs.ordinals() returns null.

Now, i don't know why lastname happens to use MultiOrdinals (or MultiDocs for that matter) to do sorting, but making MultiDocs return its MultiOrdinals instance seems to fix the issue, although I'm not sure whether the null pointer was intentional and the problem is somewhere else.

Thoughts?

Thanks a lot in advance.

[s1monw]

Thanks for opening this! This is in-fact a bug that is triggered depending on how sparse your data in that field is. I will come up with a patch very soon and I try to increase the test coverage for this to make sure we are not running into this again.

[s1monw]

thanks for reporting this so quickly!

[ghost]

Hey, thank YOU for fixing it so quickly!

ES is an amazing project, I'm just glad I can help.

--- Original Message ---

From: "Simon Willnauer" notifications@github.com Sent: August 9, 2013 9:44 AM To: "elasticsearch/elasticsearch" elasticsearch@noreply.github.com Cc: "dngdevelopment" kayowas@hotmail.com Subject: Re: [elasticsearch] NPE in BytesRefOrdValComparator (#3470)

thanks for reporting this so quickly!

---

Reply to this email directly or view it on GitHub: https://github.com/elasticsearch/elasticsearch/issues/3470#issuecomment-22391676

[elasticjava]

Many thanks Simon! that error filled our log!! +1

[tarunjangra]

Great guys, when you would be able to release this bug. We are using this very heavily. +1

[s1monw]

@tarunjangra it's coming very soon I hope we can move towards a release next week or so.

[chrisamccoy]

@s1monw Yes, we'd love a fix for this bug. We're using it heavily at Yoursports.com http://techcrunch.com/2012/10/11/sports-section-2-0-yoursports-launches-its-ambitious-project-to-build-the-facebook-of-sports/

[tarunjangra]

@s1monw i would like to share this behaviour. Query i am doing: {"query":{"match_all":{}},"sort":{"title.untouched":{"order":"desc"}}}

And result i am getting:

QueryPhaseExecutionException[[es_index][11]: query[ConstantScore(cache(org.elasticsearch.index.search.nested.NonNestedDocsFilter@953b6c6c))],from[0],size[10],sort[<custom:"title.untouched": org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource@1ca8b405>!]: Query Failed [Failed to execute main query]]; nested: NullPointerException;

Some thing interesting i just noticed. I am also getting some stats about Shards.

total: 15 successful: 10 failed: 5

But yesterday i was getting

total: 15 successful: 6 failed: 9

the question is, why these stats are changing? Is that some how usual ?

Thank you.

[s1monw]

@tarunjangra this issue depends on a lot of things if your field data implementation uses the "non-broken" code paths due to segment merges you won't see the failure on all shards so this can change over time.

[pbellora]

Is there a known workaround in the meantime? I was also using not_analyzed for sorting, and that's breaking with an NPE in BytesRefFieldComparatorSource (I'm assuming it's the same issue). I tried using keyword analysis instead but got the same failures.

[tarunjangra]

@s1monw Yep: It make sense. We are waiting for next elasticsearch release. Would like to recognise all elasticsearch team for such a great community efforts.

Thank you.

[mjebrini]

Hi I am searching with following query:

{ "query": { "filtered": { "query": { "match_all": [] }, "filter": { "and": [ { "query": { "match": { "status": "published" } } }, { "numeric_range": { "rate": { "gte": 0, "lte": 5 } } } ] } } }, "from": 1, "size": 5, "sort": [ { "activity_id": "desc" } ], "fields": [ "activity_id" ] }

After all I get into this exception:

QueryPhaseExecutionException[[activity][0]: query[ConstantScore(+QueryWrapperFilter(status:published) +rate:[0 TO 5])],from[1],size[5],sort[<custom:\"activity_id\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@bda96b>!]: Query Failed [Failed to execute main query]]; nested: ElasticSearchException[java.lang.NumberFormatException: Invalid shift value in prefixCoded bytes (is encoded value really an INT?)]; nested: UncheckedExecutionException[java.lang.NumberFormatException: Invalid shift value in prefixCoded bytes (is encoded value really an INT?)]; nested: NumberFormatException[Invalid shift value in prefixCoded bytes (is encoded value really an INT?)];

Actually this is weird, i was facing the problem at 0.90.5 .. then I upgraded my elastic search binary to 0.90.9 and I still face the same error.

[mjebrini]

I think the sorting term is the cause of the problem!