Closed imotov closed 1 year ago
I don't get a panic, but a wrong result. The lt=0
is ignored
➜ asdf curl -XPOST http://localhost:7280/api/v1/_elastic/gharchive/_search -H 'Content-Type: application/json' -d '{
"query": {
"range": {
"size": {
"gte": 15,
"lt": 0
}
}
}
}'
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"hits": [
{
"_source": {"body":"test one","size":20},
"fields": {
"body": "test one",
"size": 20
}
}
]
}
}
Here is what I just did:
quickwit
directory ran cargo clean
quickwit
directory ran RUST_BACKTRACE=1 cargo run --all-features -- run --config ../config/quickwit.yaml
{
"status": 500,
"error": {
"caused_by": null,
"reason": "Internal error: `(Internal error: `Leaf search panicked. split=01H9PBDX9S5KH9WC3FRX74MP47`., split_id: 01H9PBDX9S5KH9WC3FRX74MP47)`.",
"stack_trace": null,
"type": null
}
}
The following message showed up in the log file:
2023-09-06T22:57:23.245Z INFO quickwit_serve::index_api::rest_handler: delete-index index_id=test dry_run=false
2023-09-06T22:57:23.254Z INFO quickwit_config::index_config::serialize: Index config does not specify `index_uri`, falling back to default value. index_id=test index_uri=file:///home/igor/Projects/quickwit-oss/quickwit/quickwit/qwdata/indexes/test
2023-09-06T22:57:23.255Z INFO quickwit_serve::index_api::rest_handler: create-index index_id=test overwrite=false
2023-09-06T22:57:23.266Z INFO quickwit_metastore::metastore::metastore_event_publisher: add source test, SourceConfig { source_id: "_ingest-api-source", max_num_pipelines_per_indexer: 1, desired_num_pipelines: 1, enabled: true, source_params: IngestApi, transform_config: None, input_format: Json }
2023-09-06T22:57:23.270Z INFO quickwit_metastore::metastore::metastore_event_publisher: add source test, SourceConfig { source_id: "_ingest-cli-source", max_num_pipelines_per_indexer: 1, desired_num_pipelines: 1, enabled: true, source_params: IngestCli, transform_config: None, input_format: Json }
2023-09-06T22:57:23.275Z INFO quickwit_indexing::actors::indexing_service: Spawning indexing pipeline. pipeline_id=IndexingPipelineId { index_uid: IndexUid("test:01H9PBDW9TY2CYGA9FQEWA1J0N"), source_id: "_ingest-api-source", node_id: "leafminer", pipeline_ord: 0 }
2023-09-06T22:57:23.276Z INFO spawn_merge_pipeline{index=test gen=0}: quickwit_indexing::actors::merge_pipeline: Spawning merge pipeline. index_id=test source_id=_ingest-api-source pipeline_ord=0 root_dir=/home/igor/Projects/quickwit-oss/quickwit/quickwit/qwdata/indexing/test%01H9PBDW9TY2CYGA9FQEWA1J0N%_ingest-api-source%0%qsJa01 merge_policy=StableLogMergePolicy { config: StableLogMergePolicyConfig { min_level_num_docs: 100000, merge_factor: 10, max_merge_factor: 12, maturation_period: 172800s }, split_num_docs_target: 10000000 }
2023-09-06T22:57:23.276Z INFO spawn_pipeline{index=test gen=0}: quickwit_indexing::actors::indexing_pipeline: Spawning indexing pipeline. index_id=test source_id=_ingest-api-source pipeline_ord=0 root_dir=/home/igor/Projects/quickwit-oss/quickwit/quickwit/qwdata/indexing/test%01H9PBDW9TY2CYGA9FQEWA1J0N%_ingest-api-source%0%qsJa01
2023-09-06T22:57:23.346Z INFO quickwit_janitor::actors::delete_task_pipeline: Spawning delete tasks pipeline. index_id=test root_dir=/home/igor/Projects/quickwit-oss/quickwit/quickwit/qwdata/delete_task_service
2023-09-06T22:57:24.300Z INFO quickwit_indexing::actors::indexer: new-split split_id=01H9PBDX9S5KH9WC3FRX74MP47 partition_id=0
2023-09-06T22:57:24.301Z INFO quickwit_indexing::actors::indexer: send-to-index-serializer commit_trigger=ForceCommit split_ids=01H9PBDX9S5KH9WC3FRX74MP47 num_docs=2
2023-09-06T22:57:24.320Z INFO index-doc-batches{index_id=test source_id=_ingest-api-source pipeline_ord=0 workbench_id=01H9PBDX9R07S1B954WAKJME6H}:packager: quickwit_indexing::actors::packager: start-packaging-splits split_ids=["01H9PBDX9S5KH9WC3FRX74MP47"]
2023-09-06T22:57:24.320Z INFO index-doc-batches{index_id=test source_id=_ingest-api-source pipeline_ord=0 workbench_id=01H9PBDX9R07S1B954WAKJME6H}:packager: quickwit_indexing::actors::packager: create-packaged-split split_id="01H9PBDX9S5KH9WC3FRX74MP47"
2023-09-06T22:57:24.324Z INFO index-doc-batches{index_id=test source_id=_ingest-api-source pipeline_ord=0 workbench_id=01H9PBDX9R07S1B954WAKJME6H}:uploader: quickwit_indexing::actors::uploader: start-stage-and-store-splits split_ids=["01H9PBDX9S5KH9WC3FRX74MP47"]
2023-09-06T22:57:24.334Z INFO index-doc-batches{index_id=test source_id=_ingest-api-source pipeline_ord=0 workbench_id=01H9PBDX9R07S1B954WAKJME6H}:uploader:upload{split=01H9PBDX9S5KH9WC3FRX74MP47}:store_split: quickwit_indexing::split_store::indexing_split_store: store-split-remote-success split_size_in_megabytes=0.005806 num_docs=2 elapsed_secs=0.004687361 throughput_mb_s=1.2386501 is_mature=false
2023-09-06T22:57:24.334Z INFO index-doc-batches{index_id=test source_id=_ingest-api-source pipeline_ord=0 workbench_id=01H9PBDX9R07S1B954WAKJME6H}:uploader:upload{split=01H9PBDX9S5KH9WC3FRX74MP47}:store_split: quickwit_indexing::split_store::indexing_split_store: store-in-cache
2023-09-06T22:57:24.336Z INFO index-doc-batches{index_id=test source_id=_ingest-api-source pipeline_ord=0 workbench_id=01H9PBDX9R07S1B954WAKJME6H}:publisher{split_update=SplitsUpdate { index_id: "test", new_splits: "01H9PBDX9S5KH9WC3FRX74MP47", checkpoint_delta: Some(_ingest-api-source:∆(ingest_partition_01H7AVBKKVPTJPPS3NQ8Z4PSR8:(..00000000000000000002])) }}: quickwit_metastore::checkpoint: delta=∆(ingest_partition_01H7AVBKKVPTJPPS3NQ8Z4PSR8:(..00000000000000000002]) checkpoint=Ckpt()
2023-09-06T22:57:24.340Z INFO index-doc-batches{index_id=test source_id=_ingest-api-source pipeline_ord=0 workbench_id=01H9PBDX9R07S1B954WAKJME6H}:publisher{split_update=SplitsUpdate { index_id: "test", new_splits: "01H9PBDX9S5KH9WC3FRX74MP47", checkpoint_delta: Some(_ingest-api-source:∆(ingest_partition_01H7AVBKKVPTJPPS3NQ8Z4PSR8:(..00000000000000000002])) }}: quickwit_indexing::actors::publisher: publish-new-splits new_splits=["01H9PBDX9S5KH9WC3FRX74MP47"] checkpoint_delta=Some(_ingest-api-source:∆(ingest_partition_01H7AVBKKVPTJPPS3NQ8Z4PSR8:(..00000000000000000002]))
2023-09-06T22:57:24.340Z INFO index-doc-batches{index_id=test source_id=_ingest-api-source pipeline_ord=0 workbench_id=01H9PBDX9R07S1B954WAKJME6H}:publisher{split_update=SplitsUpdate { index_id: "test", new_splits: "01H9PBDX9S5KH9WC3FRX74MP47", checkpoint_delta: Some(_ingest-api-source:∆(ingest_partition_01H7AVBKKVPTJPPS3NQ8Z4PSR8:(..00000000000000000002])) }}: quickwit_metastore::checkpoint: delta=∆(ingest_partition_01H7AVBKKVPTJPPS3NQ8Z4PSR8:(..00000000000000000002]) checkpoint=Ckpt()
2023-09-06T22:57:24.349Z INFO root_search: quickwit_search::root: searcher_context=SearcherContext { searcher_config: SearcherConfig { aggregation_memory_limit: Byte(500000000), aggregation_bucket_limit: 65000, fast_field_cache_capacity: Byte(1000000000), split_footer_cache_capacity: Byte(500000000), partial_request_cache_capacity: Byte(64000000), max_num_concurrent_split_searches: 100, max_num_concurrent_split_streams: 100 }, leaf_search_split_semaphore: Semaphore { ll_sem: Semaphore { permits: 100 } }, split_stream_semaphore: Semaphore { ll_sem: Semaphore { permits: 100 } } } search_request=SearchRequest { index_id_patterns: ["test"], query_ast: "{\"type\":\"range\",\"field\":\"size\",\"lower_bound\":{\"Included\":15},\"upper_bound\":{\"Excluded\":0}}", start_timestamp: None, end_timestamp: None, max_hits: 10, start_offset: 0, aggregation_request: None, snippet_fields: [], sort_fields: [], scroll_ttl_secs: None }
2023-09-06T22:57:24.351Z INFO root_search:root_search_aux: quickwit_search::root: split_metadatas=[SplitMetadata { split_id: "01H9PBDX9S5KH9WC3FRX74MP47", index_uid: IndexUid("test:01H9PBDW9TY2CYGA9FQEWA1J0N"), partition_id: 0, source_id: "_ingest-api-source", node_id: "leafminer", num_docs: 2, uncompressed_docs_size_in_bytes: 63, time_range: None, create_timestamp: 1694041044, maturity: Immature { maturation_period: 172800s }, tags: {}, footer_offsets: 2253..5806, delete_opstamp: 0, num_merge_ops: 0 }]
2023-09-06T22:57:24.352Z INFO root_search:root_search_aux:search_partial_hits_phase_with_scroll:search_partial_hits_phase:leaf_search{index=["test"]}: quickwit_search::leaf: splits_num=1 split_offsets=[SplitIdAndFooterOffsets { split_id: "01H9PBDX9S5KH9WC3FRX74MP47", split_footer_start: 2253, split_footer_end: 5806, timestamp_start: None, timestamp_end: None }]
thread 'quickwit-search-0' panicked at 'attempt to subtract with overflow', /home/igor/.cargo/git/checkouts/tantivy-f70b7ea03dadae9a/49448b3/src/query/range_query/range_query_u64_fastfield.rs:121:45
stack backtrace:
0: rust_begin_unwind
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
1: core::panicking::panic_fmt
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
2: core::panicking::panic
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:117:5
3: tantivy::query::range_query::range_query_u64_fastfield::bound_to_value_range
4: <tantivy::query::range_query::range_query_u64_fastfield::FastFieldRangeWeight as tantivy::query::weight::Weight>::scorer
5: tantivy::query::weight::Weight::for_each_no_score
6: tantivy::collector::Collector::collect_segment
7: tantivy::core::searcher::Searcher::search_with_executor::{{closure}}
8: core::iter::adapters::map::map_try_fold::{{closure}}
9: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::try_fold::enumerate::{{closure}}
10: core::iter::traits::iterator::Iterator::try_fold
11: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::try_fold
12: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
13: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold
14: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next
15: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
16: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
17: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
18: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
19: core::iter::adapters::try_process
20: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
21: core::iter::traits::iterator::Iterator::collect
22: tantivy::core::executor::Executor::map
23: tantivy::core::searcher::Searcher::search_with_executor
24: tantivy::core::searcher::Searcher::search_with_statistics_provider
25: tantivy::core::searcher::Searcher::search
26: quickwit_search::leaf::leaf_search_single_split::{{closure}}::{{closure}}::{{closure}}
27: quickwit_search::thread_pool::run_cpu_intensive::{{closure}}::{{closure}}
28: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
29: std::panicking::try::do_call
30: __rust_try
31: std::panicking::try
32: std::panic::catch_unwind
33: rayon_core::unwind::halt_unwinding
34: rayon_core::registry::Registry::catch_unwind
35: rayon_core::spawn::spawn_job::{{closure}}
36: <rayon_core::job::HeapJob<BODY> as rayon_core::job::Job>::execute
37: rayon_core::job::JobRef::execute
38: rayon_core::registry::WorkerThread::execute
39: rayon_core::registry::WorkerThread::wait_until_cold
40: rayon_core::registry::WorkerThread::wait_until
41: rayon_core::registry::main_loop
42: rayon_core::registry::ThreadBuilder::run
43: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2023-09-06T22:57:24.357Z ERROR quickwit_search::thread_pool: Task running in the quickwit search pool panicked.
2023-09-06T22:57:24.359Z INFO root_search:root_search_aux:search_partial_hits_phase_with_scroll:search_partial_hits_phase:leaf_search{index=["test"]}: quickwit_search::leaf: splits_num=1 split_offsets=[SplitIdAndFooterOffsets { split_id: "01H9PBDX9S5KH9WC3FRX74MP47", split_footer_start: 2253, split_footer_end: 5806, timestamp_start: None, timestamp_end: None }]
thread 'quickwit-search-0' panicked at 'attempt to subtract with overflow', /home/igor/.cargo/git/checkouts/tantivy-f70b7ea03dadae9a/49448b3/src/query/range_query/range_query_u64_fastfield.rs:121:45
stack backtrace:
0: rust_begin_unwind
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
1: core::panicking::panic_fmt
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
2: core::panicking::panic
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:117:5
3: tantivy::query::range_query::range_query_u64_fastfield::bound_to_value_range
4: <tantivy::query::range_query::range_query_u64_fastfield::FastFieldRangeWeight as tantivy::query::weight::Weight>::scorer
5: tantivy::query::weight::Weight::for_each_no_score
6: tantivy::collector::Collector::collect_segment
7: tantivy::core::searcher::Searcher::search_with_executor::{{closure}}
8: core::iter::adapters::map::map_try_fold::{{closure}}
9: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::try_fold::enumerate::{{closure}}
10: core::iter::traits::iterator::Iterator::try_fold
11: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::try_fold
12: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
13: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold
14: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next
15: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
16: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
17: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
18: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
19: core::iter::adapters::try_process
20: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
21: core::iter::traits::iterator::Iterator::collect
22: tantivy::core::executor::Executor::map
23: tantivy::core::searcher::Searcher::search_with_executor
24: tantivy::core::searcher::Searcher::search_with_statistics_provider
25: tantivy::core::searcher::Searcher::search
26: quickwit_search::leaf::leaf_search_single_split::{{closure}}::{{closure}}::{{closure}}
27: quickwit_search::thread_pool::run_cpu_intensive::{{closure}}::{{closure}}
28: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
29: std::panicking::try::do_call
30: __rust_try
31: std::panicking::try
32: std::panic::catch_unwind
33: rayon_core::unwind::halt_unwinding
34: rayon_core::registry::Registry::catch_unwind
35: rayon_core::spawn::spawn_job::{{closure}}
36: <rayon_core::job::HeapJob<BODY> as rayon_core::job::Job>::execute
37: rayon_core::job::JobRef::execute
38: rayon_core::registry::WorkerThread::execute
39: rayon_core::registry::WorkerThread::wait_until_cold
40: rayon_core::registry::WorkerThread::wait_until
41: rayon_core::registry::main_loop
42: rayon_core::registry::ThreadBuilder::run
43: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2023-09-06T22:57:24.364Z ERROR quickwit_search::thread_pool: Task running in the quickwit search pool panicked.
2023-09-06T22:57:24.366Z ERROR root_search:root_search_aux:search_partial_hits_phase_with_scroll:search_partial_hits_phase: quickwit_search::root: Leaf search response contains at least one failed split. failed_splits=[SplitSearchError { error: "Internal error: `Leaf search panicked. split=01H9PBDX9S5KH9WC3FRX74MP47`.", split_id: "01H9PBDX9S5KH9WC3FRX74MP47", retryable_error: true }]
The machine is
⚡igor@leafminer:~/Downloads❯ uname -a
Linux leafminer 6.2.0-32-generic #32~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 18 10:40:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Ah okay, the difference is the debug mode, which adds overflow checks.
Describe the bug
An ES-compatible range query with the lower bound higher then the upper bound causes the searcher to panic.
Steps to reproduce (if applicable)
Expected behavior The query should return no results.
Configuration: Current main branch.