search

blevesearch / bleve

A modern text/numeric/geo-spatial/vector indexing library for go
Apache License 2.0
10.05k stars 680 forks source link

runtime error: nil pointer dereference in roaring.go", line 667, in (*Bitmap).GetCardinality #1662

Open agnivade opened 2 years ago

agnivade commented 2 years ago

Sorry to bother you with more issues. We have another Sentry crash, this time in the roaring bitmap library, but it gets called from the zap library.

runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "github.com/RoaringBitmap/roaring@v0.9.4/roaring.go", line 667, in (*Bitmap).GetCardinality
  File "github.com/blevesearch/zapx/v15@v15.3.2/posting.go", line 239, in (*PostingsList).Count
  File "github.com/blevesearch/bleve/v2@v2.3.0/index/scorch/snapshot_index_tfr.go", line 177, in (*IndexSnapshotTermFieldReader).Count
  File "github.com/blevesearch/bleve/v2@v2.3.0/search/searcher/search_term.go", line 76, in (*TermSearcher).Count
  File "github.com/blevesearch/bleve/v2@v2.3.0/search/searcher/search_conjunction.go", line 259, in (*ConjunctionSearcher).Count
  File "github.com/blevesearch/bleve/v2@v2.3.0/search/searcher/search_conjunction.go", line 259, in (*ConjunctionSearcher).Count
  File "github.com/blevesearch/bleve/v2@v2.3.0/search/searcher/search_boolean.go", line 403, in (*BooleanSearcher).Count
  File "github.com/blevesearch/bleve/v2@v2.3.0/search/searcher/ordered_searchers_list.go", line 30, in OrderedSearcherList.Less
  File "sort/sort.go", line 40, in insertionSort
  File "sort/sort.go", line 222, in quickSort
  File "sort/sort.go", line 231, in Sort
  File "github.com/blevesearch/bleve/v2@v2.3.0/search/searcher/search_conjunction.go", line 54, in NewConjunctionSearcher
  File "github.com/blevesearch/bleve/v2@v2.3.0/search/query/conjunction.go", line 78, in (*ConjunctionQuery).Searcher
  File "github.com/blevesearch/bleve/v2@v2.3.0/search/query/boolean.go", line 132, in (*BooleanQuery).Searcher
  File "github.com/blevesearch/bleve/v2@v2.3.0/index_impl.go", line 471, in (*indexImpl).SearchInContext
  File "github.com/blevesearch/bleve/v2@v2.3.0/index_impl.go", line 368, in (*indexImpl).Search
  File "github.com/mattermost/mattermost-server/v6/services/searchengine/bleveengine/search.go", line 215, in (*BleveEngine).SearchPosts
  File "github.com/mattermost/mattermost-server/v6/store/searchlayer/post_layer.go", line 150, in SearchPostStore.searchPostsForUserByEngine
  File "github.com/mattermost/mattermost-server/v6/store/searchlayer/post_layer.go", line 176, in SearchPostStore.SearchPostsForUser
  File "github.com/mattermost/mattermost-server/v6/store/timerlayer/timerlayer.go", line 5295, in (*TimerLayerPostStore).SearchPostsForUser
  File "github.com/mattermost/mattermost-server/v6/app/post.go", line 1322, in (*App).SearchPostsForUser
  File "github.com/mattermost/mattermost-server/v6/api4/post.go", line 602, in searchPosts
  File "github.com/mattermost/mattermost-server/v6/api4/post.go", line 555, in searchPostsInTeam
  File "github.com/mattermost/mattermost-server/v6/web/handlers.go", line 339, in Handler.ServeHTTP
  File "github.com/mattermost/gziphandler@v0.0.1/gzip.go", line 337, in GzipHandlerWithOpts.func1.1
  File "net/http/server.go", line 2049, in HandlerFunc.ServeHTTP
  File "github.com/gorilla/mux@v1.8.0/mux.go", line 210, in (*Router).ServeHTTP
  File "net/http/server.go", line 2049, in HandlerFunc.ServeHTTP
  File "net/http/server.go", line 2867, in serverHandler.ServeHTTP
  File "net/http/server.go", line 1932, in (*conn).serve

Looking at the exact line of error, it's here: https://github.com/RoaringBitmap/roaring/blob/e6471ed95f1d549fe1b284310217906a9d62035e/roaring.go#L667. This seems a race condition, or somehow a nil container was passed.

agnivade commented 1 year ago

Hi, any progress on this? We still receive sentry crashes for this.

jgschis commented 1 year ago

This happens to me too sometimes, I will look into this as I thought it was just me.

jgschis commented 1 year ago

I stopped using the min property of the disjunction query, and then this error stopped.

however, it's started happening again.

So again I am looking into it.

agarciamontoro commented 8 months ago

We are seeing several crash reports, either equal to the one reported here, or very similar, using Bleve 2.3.8. Some end up crashing in zapx, others in RoaringBitmap:

runtime.boundsError: runtime error: index out of range [4] with length 0
  File "github.com/RoaringBitmap/roaring@v1.3.0/roaringarray.go", line 347, in (*roaringArray).getKeyAtIndex
  File "github.com/RoaringBitmap/roaring@v1.3.0/roaring.go", line 1382, in AndNot
  File "github.com/blevesearch/zapx/v15@v15.3.11/posting.go", line 224, in (*PostingsList).iterator
  File "github.com/blevesearch/zapx/v15@v15.3.11/posting.go", line 155, in (*PostingsList).Iterator
  File "github.com/blevesearch/bleve/v2@v2.3.8/index/scorch/snapshot_index.go", line 582, in (*IndexSnapshot).TermFieldReader
  File "github.com/blevesearch/bleve/v2@v2.3.8/index/scorch/snapshot_index_tfr.go", line 149, in (*IndexSnapshotTermFieldReader).Advance
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_term.go", line 106, in (*TermSearcher).Advance
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 253, in (*ConjunctionSearcher).advanceChild
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 200, in (*ConjunctionSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 218, in (*ConjunctionSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 165, in (*BooleanSearcher).advanceNextMust
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 316, in (*BooleanSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 396, in (*BooleanSearcher).Advance
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 253, in (*ConjunctionSearcher).advanceChild
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 200, in (*ConjunctionSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 165, in (*BooleanSearcher).advanceNextMust
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 316, in (*BooleanSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/collector/topn.go", line 205, in (*TopNCollector).Collect
  File "github.com/blevesearch/bleve/v2@v2.3.8/index_impl.go", line 551, in (*indexImpl).SearchInContext
  File "github.com/blevesearch/bleve/v2@v2.3.8/index_impl.go", line 369, in (*indexImpl).Search
  runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "github.com/RoaringBitmap/roaring@v1.3.0/roaring.go", line 810, in (*Bitmap).GetCardinality
  File "github.com/blevesearch/zapx/v15@v15.3.11/posting.go", line 243, in (*PostingsList).Count
  File "github.com/blevesearch/bleve/v2@v2.3.8/index/scorch/snapshot_index_tfr.go", line 194, in (*IndexSnapshotTermFieldReader).Count
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_term.go", line 77, in (*TermSearcher).Count
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/ordered_searchers_list.go", line 30, in OrderedSearcherList.Less
  File "sort/zsortinterface.go", line 12, in insertionSort
  File "sort/zsortinterface.go", line 73, in pdqsort
  File "sort/sort.go", line 48, in Sort
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 56, in NewConjunctionSearcher
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/query/conjunction.go", line 79, in (*ConjunctionQuery).Searcher
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/query/match.go", line 169, in (*MatchQuery).Searcher
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/query/conjunction.go", line 59, in (*ConjunctionQuery).Searcher
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/query/boolean.go", line 133, in (*BooleanQuery).Searcher
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/query/conjunction.go", line 59, in (*ConjunctionQuery).Searcher
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/query/boolean.go", line 133, in (*BooleanQuery).Searcher
  File "github.com/blevesearch/bleve/v2@v2.3.8/index_impl.go", line 485, in (*indexImpl).SearchInContext
  File "github.com/blevesearch/bleve/v2@v2.3.8/index_impl.go", line 369, in (*indexImpl).Search
  runtime.errorString: runtime error: integer divide by zero
  File "github.com/blevesearch/zapx/v15@v15.3.11/posting.go", line 644, in (*PostingsIterator).nextDocNumAtOrAfter
  File "github.com/blevesearch/zapx/v15@v15.3.11/posting.go", line 537, in (*PostingsIterator).nextAtOrAfter
  File "github.com/blevesearch/zapx/v15@v15.3.11/posting.go", line 526, in (*PostingsIterator).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/index/scorch/snapshot_index_tfr.go", line 88, in (*IndexSnapshotTermFieldReader).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_term.go", line 89, in (*TermSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 218, in (*ConjunctionSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 218, in (*ConjunctionSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 246, in (*ConjunctionSearcher).Advance
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 356, in (*BooleanSearcher).Advance
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 253, in (*ConjunctionSearcher).advanceChild
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 190, in (*ConjunctionSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 165, in (*BooleanSearcher).advanceNextMust
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 316, in (*BooleanSearcher).Next
  File "github.com/blevesearch/bleve/v2@v2.3.8/search/collector/topn.go", line 226, in (*TopNCollector).Collect
  File "github.com/blevesearch/bleve/v2@v2.3.8/index_impl.go", line 551, in (*indexImpl).SearchInContext
  File "github.com/blevesearch/bleve/v2@v2.3.8/index_impl.go", line 369, in (*indexImpl).Search

These come from Sentry, so I don't have much more data, hope it's somewhat useful.

jgschis commented 8 months ago

I still get the same issue in Couchbase. I have talked to the team about this and gave them an application to reproduce it, and they did reproduce it. But not sure what's come of it since then (almost a year ago).

cpoile commented 1 month ago

We received another crash, bleve -> zapx -> RoaringBitmap, this time in AndCardinality:

runtime.errorString: runtime error: invalid memory address or nil pointer dereference
github.com/RoaringBitmap/roaring@v1.6.0/roaring.go in (*Bitmap).AndCardinality at line 996
github.com/blevesearch/zapx/v15@v15.3.13/posting.go in (*PostingsList).Count at line 245
github.com/blevesearch/bleve/v2@v2.3.10/index/scorch/snapshot_index_tfr.go in (*IndexSnapshotTermFieldReader).Count at line 194
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_term.go in newTermSearcherFromReader at line 64
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_term.go in NewTermSearcherBytes at line 54
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_term.go in NewTermSearcher at line 45
github.com/blevesearch/bleve/v2@v2.3.10/search/query/term.go in (*TermQuery).Searcher at line 62
github.com/blevesearch/bleve/v2@v2.3.10/search/query/conjunction.go in (*ConjunctionQuery).Searcher at line 60
github.com/blevesearch/bleve/v2@v2.3.10/search/query/match.go in (*MatchQuery).Searcher at line 169
github.com/blevesearch/bleve/v2@v2.3.10/search/query/conjunction.go in (*ConjunctionQuery).Searcher at line 60
github.com/blevesearch/bleve/v2@v2.3.10/search/query/boolean.go in (*BooleanQuery).Searcher at line 134
github.com/blevesearch/bleve/v2@v2.3.10/search/query/conjunction.go in (*ConjunctionQuery).Searcher at line 60
github.com/blevesearch/bleve/v2@v2.3.10/search/query/boolean.go in (*BooleanQuery).Searcher at line 134
github.com/blevesearch/bleve/v2@v2.3.10/index_impl.go in (*indexImpl).SearchInContext at line 499
github.com/blevesearch/bleve/v2@v2.3.10/index_impl.go in (*indexImpl).Search at line 371
cpoile commented 1 month ago

We've received a couple more of these. The first:

runtime.errorString: runtime error: integer divide by zero
File "github.com/blevesearch/zapx/v15@v15.3.11/posting.go", line 644, in (*PostingsIterator).nextDocNumAtOrAfter
File "github.com/blevesearch/zapx/v15@v15.3.11/posting.go", line 537, in (*PostingsIterator).nextAtOrAfter
File "github.com/blevesearch/zapx/v15@v15.3.11/posting.go", line 526, in (*PostingsIterator).Next
File "github.com/blevesearch/bleve/v2@v2.3.8/index/scorch/snapshot_index_tfr.go", line 88, in (*IndexSnapshotTermFieldReader).Next
File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_term.go", line 89, in (*TermSearcher).Next
File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 218, in (*ConjunctionSearcher).Next
File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 218, in (*ConjunctionSearcher).Next
File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 246, in (*ConjunctionSearcher).Advance
File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 356, in (*BooleanSearcher).Advance
File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 253, in (*ConjunctionSearcher).advanceChild
File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_conjunction.go", line 190, in (*ConjunctionSearcher).Next
File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 165, in (*BooleanSearcher).advanceNextMust
File "github.com/blevesearch/bleve/v2@v2.3.8/search/searcher/search_boolean.go", line 316, in (*BooleanSearcher).Next
File "github.com/blevesearch/bleve/v2@v2.3.8/search/collector/topn.go", line 226, in (*TopNCollector).Collect
File "github.com/blevesearch/bleve/v2@v2.3.8/index_impl.go", line 551, in (*indexImpl).SearchInContext
File "github.com/blevesearch/bleve/v2@v2.3.8/index_impl.go", line 369, in (*indexImpl).Search

The second:

runtime error: integer divide by zero
github.com/blevesearch/zapx/v15@v15.3.13/posting.go in (*PostingsIterator).nextDocNumAtOrAfterClean at line 747
github.com/blevesearch/zapx/v15@v15.3.13/posting.go in (*PostingsIterator).nextDocNumAtOrAfter at line 632
github.com/blevesearch/zapx/v15@v15.3.13/posting.go in (*PostingsIterator).nextAtOrAfter at line 537
github.com/blevesearch/zapx/v15@v15.3.13/posting.go in (*PostingsIterator).Advance at line 532
github.com/blevesearch/bleve/v2@v2.3.10/index/scorch/snapshot_index_tfr.go in (*IndexSnapshotTermFieldReader).Advance at line 169
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_term.go in (*TermSearcher).Advance at line 109
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_conjunction.go in (*ConjunctionSearcher).advanceChild at line 253
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_conjunction.go in (*ConjunctionSearcher).Next at line 200
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_conjunction.go in (*ConjunctionSearcher).Next at line 218
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_boolean.go in (*BooleanSearcher).advanceNextMust at line 165
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_boolean.go in (*BooleanSearcher).Next at line 316
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_boolean.go in (*BooleanSearcher).Advance at line 396
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_conjunction.go in (*ConjunctionSearcher).advanceChild at line 253
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_conjunction.go in (*ConjunctionSearcher).Next at line 190
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_boolean.go in (*BooleanSearcher).advanceNextMust at line 165
github.com/blevesearch/bleve/v2@v2.3.10/search/searcher/search_boolean.go in (*BooleanSearcher).Next at line 316
github.com/blevesearch/bleve/v2@v2.3.10/search/collector/topn.go in (*TopNCollector).Collect at line 228
github.com/blevesearch/bleve/v2@v2.3.10/index_impl.go in (*indexImpl).SearchInContext at line 580
github.com/blevesearch/bleve/v2@v2.3.10/index_impl.go in (*indexImpl).Search at line 371

Hope that helps?