search

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
8.21k stars 336 forks source link

Search stream api errors out after field mapping change #1689

Closed leongshengmin closed 2 years ago

leongshengmin commented 2 years ago

Describe the bug After index mapping type change, search stream API errors out.

# clickhouse query
select count() from url('localhost:8080/api/v1/my_idx/search/stream?query=app_name:my_app&fast_field=app_name&output_format=click_house_row_binary', RowBinary, 'app_name String')

# QW error log
Jun 30 03:48:41 ip-10-189-129-185 quickwit[24963]: 2022-06-30T03:48:41.447Z ERROR quickwit_serve::search_api::rest_handler: Error when streaming search results. error=InternalError("Internal error: `Mixed types (u64, i64) for fast field and partition field are not supported.`.")
Jun 30 05:39:51 ip-10-189-129-185 quickwit[24963]: 2022-06-30T05:39:51.539Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed

Steps to reproduce (if applicable) Steps to reproduce the behavior:

  1. update index field mapping e.g. change timestamp field from u64 to i64
  2. index more data
  3. call search stream api from clickhouse.

Expected behavior No error expected.

Configuration: Please provide:

  1. Output of quickwit --version: Quickwit 0.3.1-nightly (x86_64-unknown-linux-gnu 190d6cd71 2022-06-27)
  2. The index_config.yaml
fmassot commented 2 years ago

Hi @leongshengmin, thanks for the report.

The first major issue for you is that... Quickwit does not support yet changes in the doc mapping. If you change the mapping, you have to delete the index, create a new one and index data.

The second issue is that we currently don't support mixed types (u64, i64) for fast field and partition field as indicated in the log. This can be easily fixed. But with your query example /api/v1/my_idx/search/stream?query=app_name:my_app&fast_field=app_name&output_format=click_house_row_binary it should not happen because you did not specify a partition field.

Can you confirm that you put the query param partition_by_field to generate such an error?

leongshengmin commented 2 years ago

hi @fmassot thanks for the quick reply, have confirmed that am still seeing the same error without specifying the partition_by_field query param

fmassot commented 2 years ago

Ok, this is super strange :/

Can you provide more logs from the searcher? I made a test and here is what I have without the partition_by_field query param:

2022-06-30T09:03:41.232Z  INFO quickwit_serve::search_api::rest_handler: search_stream index_id=hdfs-logs request=SearchStreamRequestQueryString { query: "hdfs", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "tenant_id", output_format: ClickHouseRowBinary, partition_by_field: None }
2022-06-30T09:03:41.233Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hdfs-logs" splits=[SplitIdAndFooterOffsets { split_id: "01G6SX990KDCGE90K8SK7BEF41", split_footer_start: 808631, split_footer_end: 814099 }, SplitIdAndFooterOffsets { split_id: "01G6SXF7EE1SN0QPXKH4GW3VK9", split_footer_start: 808634, split_footer_end: 814134 }]

I want to check particularly the first line where I can see that partition_by_field is None.

leongshengmin commented 2 years ago

Sure these are the logs

Jun 30 08:14:51 ip-10-189-129-185 quickwit[24963]: 2022-06-30T08:14:51.685Z  INFO quickwit_serve::search_api::rest_handler: search_stream index_id=my_idx request=SearchStreamRequestQueryString { query: "app_name:my_app", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "app_name", output_format: ClickHouseRowBinary, partition_by_field: None }
Jun 30 08:14:51 ip-10-189-129-185 quickwit[24963]: 2022-06-30T08:14:51.817Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="grab" splits=[SplitIdAndFooterOffsets { split_id: "01G6MW3Q370CCCEE86YQH9C5BM", split_footer_start: 4341838823, split_footer_end: 4342221670 }, SplitIdAndFooterOffsets { split_id: "01G6MX8RM1QE2CVXXW4Y3T83D2", split_footer_start: 3828641765, split_footer_end: 3829057298 }, SplitIdAndFooterOffsets { split_id: "01G6MXS34YNH5AVNHP9MWEH8QS", split_footer_start: 1981272768, split_footer_end: 1981516268 }, SplitIdAndFooterOffsets { split_id: "01G6MY80222E5DV65SQRJJ79JV", split_footer_start: 2538610864, split_footer_end: 2538890359 }, SplitIdAndFooterOffsets { split_id: "01G6MYK20TS8P9QHVTT5MFG4MX", split_footer_start: 2433842269, split_footer_end: 2434101953 }, SplitIdAndFooterOffsets { split_id: "01G6MYY307GCYF5MXEQJCRMTPT", split_footer_start: 2394713447, split_footer_end: 2394971408 }, SplitIdAndFooterOffsets { split_id: "01G6MZ958A8RJM3ZZK4RTX9EAA", split_footer_start: 2371375034, split_footer_end: 2371640066 }, SplitIdAndFooterOffsets { split_id: "01G6MZKV98BYA92DTZPKAECMXA", split_footer_start: 415228357, split_footer_end: 415291041 }, SplitIdAndFooterOffsets { split_id: "01G6MZM63NQ8QR67DB1Q54AKYP", split_footer_start: 2355373354, split_footer_end: 2355627748 }, SplitIdAndFooterOffsets { split_id: "01G6MZQYBDFZ044B64HPK86N8B", split_footer_start: 431290117, split_footer_end: 431356533 }]
Jun 30 08:14:51 ip-10-189-129-185 quickwit[24963]: 2022-06-30T08:14:51.876Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="grab" splits=[SplitIdAndFooterOffsets { split_id: "01G6MW3Q370CCCEE86YQH9C5BM", split_footer_start: 4341838823, split_footer_end: 4342221670 }, SplitIdAndFooterOffsets { split_id: "01G6MX8RM1QE2CVXXW4Y3T83D2", split_footer_start: 3828641765, split_footer_end: 3829057298 }, SplitIdAndFooterOffsets { split_id: "01G6MXS34YNH5AVNHP9MWEH8QS", split_footer_start: 1981272768, split_footer_end: 1981516268 }, SplitIdAndFooterOffsets { split_id: "01G6MY80222E5DV65SQRJJ79JV", split_footer_start: 2538610864, split_footer_end: 2538890359 }, SplitIdAndFooterOffsets { split_id: "01G6MYK20TS8P9QHVTT5MFG4MX", split_footer_start: 2433842269, split_footer_end: 2434101953 }, SplitIdAndFooterOffsets { split_id: "01G6MYY307GCYF5MXEQJCRMTPT", split_footer_start: 2394713447, split_footer_end: 2394971408 }, SplitIdAndFooterOffsets { split_id: "01G6MZ958A8RJM3ZZK4RTX9EAA", split_footer_start: 2371375034, split_footer_end: 2371640066 }, SplitIdAndFooterOffsets { split_id: "01G6MZKV98BYA92DTZPKAECMXA", split_footer_start: 415228357, split_footer_end: 415291041 }, SplitIdAndFooterOffsets { split_id: "01G6MZM63NQ8QR67DB1Q54AKYP", split_footer_start: 2355373354, split_footer_end: 2355627748 }, SplitIdAndFooterOffsets { split_id: "01G6MZQYBDFZ044B64HPK86N8B", split_footer_start: 431290117, split_footer_end: 431356533 }]
Jun 30 08:14:51 ip-10-189-129-185 quickwit[24963]: 2022-06-30T08:14:51.879Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
Jun 30 08:14:51 ip-10-189-129-185 quickwit[24963]: 2022-06-30T08:14:51.927Z ERROR quickwit_serve::search_api::rest_handler: Error when streaming search results. error=InternalError("Internal error: `Mixed types (u64, i64) for fast field and partition field are not supported.`.")
fmassot commented 2 years ago

Thanks... still weird. Can you please print logs with debug level? Just start a searcher like this: RUST_LOG=quickwit=debug ./quickwit run --service searcher

Can you share also your index config?

leongshengmin commented 2 years ago

Sure, Debug logs:

Jun 30 10:34:09 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:09.752Z DEBUG quickwit_search::cluster_client: Leaf search stream response error. Retry once to execute LeafSearchStreamRequest { request: Some(SearchStreamRequest { index_id: "my_idx", query: "app_name:my_app", search_fields: [], start_timestamp: None, end_timestamp: None, fast_field: "app_name", output_format: ClickHouseRowBinary, partition_by_field: None }), split_offsets: [SplitIdAndFooterOffsets { split_id: "01G6MW3Q370CCCEE86YQH9C5BM", split_footer_start: 4341838823, split_footer_end: 4342221670 }, SplitIdAndFooterOffsets { split_id: "01G6MX8RM1QE2CVXXW4Y3T83D2", split_footer_start: 3828641765, split_footer_end: 3829057298 }, SplitIdAndFooterOffsets { split_id: "01G6MXS34YNH5AVNHP9MWEH8QS", split_footer_start: 1981272768, split_footer_end: 1981516268 }, SplitIdAndFooterOffsets { split_id: "01G6MY80222E5DV65SQRJJ79JV", split_footer_start: 2538610864, split_footer_end: 2538890359 }, SplitIdAndFooterOffsets { split_id: "01G6MYK20TS8P9QHVTT5MFG4MX", split_footer_start: 2433842269, split_footer_end: 2434101953 }, SplitIdAndFooterOffsets { split_id: "01G6MYY307GCYF5MXEQJCRMTPT", split_footer_start: 2394713447, split_footer_end: 2394971408 }, SplitIdAndFooterOffsets { split_id: "01G6MZ958A8RJM3ZZK4RTX9EAA", split_footer_start: 2371375034, split_footer_end: 2371640066 }, SplitIdAndFooterOffsets { split_id: "01G6MZKV98BYA92DTZPKAECMXA", split_footer_start: 415228357, split_footer_end: 415291041 }, SplitIdAndFooterOffsets { split_id: "01G6MZM63NQ8QR67DB1Q54AKYP", split_footer_start: 2355373354, split_footer_end: 2355627748 }, SplitIdAndFooterOffsets { split_id: "01G6MZQYBDFZ044B64HPK86N8B", split_footer_start: 431290117, split_footer_end: 431356533 }], doc_mapper: "{\"type\":\"default\",\"store_source\":false,\"default_search_fields\":[\"app_name\",\"message\"],\"timestamp_field\":\"timestamp\",\"field_mappings\":[{\"name\":\"timestamp\",\"type\":\"i64\",\"fast\":true,\"indexed\":true,\"stored\":true},{\"name\":\"log_level\",\"type\":\"text\",\"fast\":false,\"fieldnorms\":false,\"indexed\":true,\"record\":\"basic\",\"stored\":true,\"tokenizer\":\"raw\"},{\"name\":\"app_name\",\"type\":\"text\",\"fast\":true,\"fieldnorms\":false,\"indexed\":true,\"record\":\"basic\",\"stored\":true,\"tokenizer\":\"raw\"},{\"name\":\"message\",\"type\":\"text\",\"fast\":false,\"fieldnorms\":false,\"indexed\":true,\"record\":\"position\",\"stored\":true,\"tokenizer\":\"default\"}],\"tag_fields\":[\"app_name\"],\"mode\":\"lenient\",\"dynamic_mapping\":null}", index_uri: "s3://dev-quickwit-poc/indexes/my_idx" } with Grpc(0.0.0.0:7281)
Jun 30 10:34:09 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:09.753Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="my_idx" splits=[SplitIdAndFooterOffsets { split_id: "01G6MW3Q370CCCEE86YQH9C5BM", split_footer_start: 4341838823, split_footer_end: 4342221670 }, SplitIdAndFooterOffsets { split_id: "01G6MX8RM1QE2CVXXW4Y3T83D2", split_footer_start: 3828641765, split_footer_end: 3829057298 }, SplitIdAndFooterOffsets { split_id: "01G6MXS34YNH5AVNHP9MWEH8QS", split_footer_start: 1981272768, split_footer_end: 1981516268 }, SplitIdAndFooterOffsets { split_id: "01G6MY80222E5DV65SQRJJ79JV", split_footer_start: 2538610864, split_footer_end: 2538890359 }, SplitIdAndFooterOffsets { split_id: "01G6MYK20TS8P9QHVTT5MFG4MX", split_footer_start: 2433842269, split_footer_end: 2434101953 }, SplitIdAndFooterOffsets { split_id: "01G6MYY307GCYF5MXEQJCRMTPT", split_footer_start: 2394713447, split_footer_end: 2394971408 }, SplitIdAndFooterOffsets { split_id: "01G6MZ958A8RJM3ZZK4RTX9EAA", split_footer_start: 2371375034, split_footer_end: 2371640066 }, SplitIdAndFooterOffsets { split_id: "01G6MZKV98BYA92DTZPKAECMXA", split_footer_start: 415228357, split_footer_end: 415291041 }, SplitIdAndFooterOffsets { split_id: "01G6MZM63NQ8QR67DB1Q54AKYP", split_footer_start: 2355373354, split_footer_end: 2355627748 }, SplitIdAndFooterOffsets { split_id: "01G6MZQYBDFZ044B64HPK86N8B", split_footer_start: 431290117, split_footer_end: 431356533 }]
Jun 30 10:34:09 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:09.769Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
Jun 30 10:34:09 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:09.823Z ERROR quickwit_serve::search_api::rest_handler: Error when streaming search results. error=InternalError("Internal error: `Mixed types (u64, i64) for fast field and partition field are not supported.`.")
Jun 30 10:34:09 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:09.835Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
Jun 30 10:34:19 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:19.826Z  INFO quickwit_serve::search_api::rest_handler: search_stream index_id=my_idx request=SearchStreamRequestQueryString { query: "app_name:my_app", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "app_name", output_format: ClickHouseRowBinary, partition_by_field: None }
Jun 30 10:34:19 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:19.884Z DEBUG root_search_stream{search_stream_request=SearchStreamRequest { index_id: "my_idx", query: "app_name:my_app", search_fields: [], start_timestamp: None, end_timestamp: None, fast_field: "app_name", output_format: ClickHouseRowBinary, partition_by_field: None }}:index_metadata{index_id="my_idx"}: quickwit_metastore::metastore::postgresql_metastore: commit
Jun 30 10:34:19 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:19.959Z DEBUG root_search_stream{search_stream_request=SearchStreamRequest { index_id: "my_idx", query: "app_name:my_app", search_fields: [], start_timestamp: None, end_timestamp: None, fast_field: "app_name", output_format: ClickHouseRowBinary, partition_by_field: None }}:list_splits{index_id="my_idx" state=Published time_range_opt=None tags=Some(Or([Tag { is_present: false, tag: "app_name!" }, Tag { is_present: true, tag: "app_name:my_app" }]))}: quickwit_metastore::metastore::postgresql_metastore: commit
Jun 30 10:34:19 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:19.960Z DEBUG root_search_stream{search_stream_request=SearchStreamRequest { index_id: "my_idx", query: "app_name:my_app", search_fields: [], start_timestamp: None, end_timestamp: None, fast_field: "app_name", output_format: ClickHouseRowBinary, partition_by_field: None }}: quickwit_search::search_stream::root: Assigned leaf search jobs. assigned_leaf_search_jobs=[(Grpc(0.0.0.0:7281), [SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MW3Q370CCCEE86YQH9C5BM", split_footer_start: 4341838823, split_footer_end: 4342221670 } }, SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MX8RM1QE2CVXXW4Y3T83D2", split_footer_start: 3828641765, split_footer_end: 3829057298 } }, SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MXS34YNH5AVNHP9MWEH8QS", split_footer_start: 1981272768, split_footer_end: 1981516268 } }, SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MY80222E5DV65SQRJJ79JV", split_footer_start: 2538610864, split_footer_end: 2538890359 } }, SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MYK20TS8P9QHVTT5MFG4MX", split_footer_start: 2433842269, split_footer_end: 2434101953 } }, SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MYY307GCYF5MXEQJCRMTPT", split_footer_start: 2394713447, split_footer_end: 2394971408 } }, SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MZ958A8RJM3ZZK4RTX9EAA", split_footer_start: 2371375034, split_footer_end: 2371640066 } }, SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MZKV98BYA92DTZPKAECMXA", split_footer_start: 415228357, split_footer_end: 415291041 } }, SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MZM63NQ8QR67DB1Q54AKYP", split_footer_start: 2355373354, split_footer_end: 2355627748 } }, SearchJob { cost: 1, offsets: SplitIdAndFooterOffsets { split_id: "01G6MZQYBDFZ044B64HPK86N8B", split_footer_start: 431290117, split_footer_end: 431356533 } }])]
Jun 30 10:34:19 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:19.961Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="my_idx" splits=[SplitIdAndFooterOffsets { split_id: "01G6MW3Q370CCCEE86YQH9C5BM", split_footer_start: 4341838823, split_footer_end: 4342221670 }, SplitIdAndFooterOffsets { split_id: "01G6MX8RM1QE2CVXXW4Y3T83D2", split_footer_start: 3828641765, split_footer_end: 3829057298 }, SplitIdAndFooterOffsets { split_id: "01G6MXS34YNH5AVNHP9MWEH8QS", split_footer_start: 1981272768, split_footer_end: 1981516268 }, SplitIdAndFooterOffsets { split_id: "01G6MY80222E5DV65SQRJJ79JV", split_footer_start: 2538610864, split_footer_end: 2538890359 }, SplitIdAndFooterOffsets { split_id: "01G6MYK20TS8P9QHVTT5MFG4MX", split_footer_start: 2433842269, split_footer_end: 2434101953 }, SplitIdAndFooterOffsets { split_id: "01G6MYY307GCYF5MXEQJCRMTPT", split_footer_start: 2394713447, split_footer_end: 2394971408 }, SplitIdAndFooterOffsets { split_id: "01G6MZ958A8RJM3ZZK4RTX9EAA", split_footer_start: 2371375034, split_footer_end: 2371640066 }, SplitIdAndFooterOffsets { split_id: "01G6MZKV98BYA92DTZPKAECMXA", split_footer_start: 415228357, split_footer_end: 415291041 }, SplitIdAndFooterOffsets { split_id: "01G6MZM63NQ8QR67DB1Q54AKYP", split_footer_start: 2355373354, split_footer_end: 2355627748 }, SplitIdAndFooterOffsets { split_id: "01G6MZQYBDFZ044B64HPK86N8B", split_footer_start: 431290117, split_footer_end: 431356533 }]
Jun 30 10:34:20 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:20.041Z DEBUG quickwit_search::cluster_client: Leaf search stream response error. Retry once to execute LeafSearchStreamRequest { request: Some(SearchStreamRequest { index_id: "my_idx", query: "app_name:my_app", search_fields: [], start_timestamp: None, end_timestamp: None, fast_field: "app_name", output_format: ClickHouseRowBinary, partition_by_field: None }), split_offsets: [SplitIdAndFooterOffsets { split_id: "01G6MW3Q370CCCEE86YQH9C5BM", split_footer_start: 4341838823, split_footer_end: 4342221670 }, SplitIdAndFooterOffsets { split_id: "01G6MX8RM1QE2CVXXW4Y3T83D2", split_footer_start: 3828641765, split_footer_end: 3829057298 }, SplitIdAndFooterOffsets { split_id: "01G6MXS34YNH5AVNHP9MWEH8QS", split_footer_start: 1981272768, split_footer_end: 1981516268 }, SplitIdAndFooterOffsets { split_id: "01G6MY80222E5DV65SQRJJ79JV", split_footer_start: 2538610864, split_footer_end: 2538890359 }, SplitIdAndFooterOffsets { split_id: "01G6MYK20TS8P9QHVTT5MFG4MX", split_footer_start: 2433842269, split_footer_end: 2434101953 }, SplitIdAndFooterOffsets { split_id: "01G6MYY307GCYF5MXEQJCRMTPT", split_footer_start: 2394713447, split_footer_end: 2394971408 }, SplitIdAndFooterOffsets { split_id: "01G6MZ958A8RJM3ZZK4RTX9EAA", split_footer_start: 2371375034, split_footer_end: 2371640066 }, SplitIdAndFooterOffsets { split_id: "01G6MZKV98BYA92DTZPKAECMXA", split_footer_start: 415228357, split_footer_end: 415291041 }, SplitIdAndFooterOffsets { split_id: "01G6MZM63NQ8QR67DB1Q54AKYP", split_footer_start: 2355373354, split_footer_end: 2355627748 }, SplitIdAndFooterOffsets { split_id: "01G6MZQYBDFZ044B64HPK86N8B", split_footer_start: 431290117, split_footer_end: 431356533 }], doc_mapper: "{\"type\":\"default\",\"store_source\":false,\"default_search_fields\":[\"app_name\",\"message\"],\"timestamp_field\":\"timestamp\",\"field_mappings\":[{\"name\":\"timestamp\",\"type\":\"i64\",\"fast\":true,\"indexed\":true,\"stored\":true},{\"name\":\"log_level\",\"type\":\"text\",\"fast\":false,\"fieldnorms\":false,\"indexed\":true,\"record\":\"basic\",\"stored\":true,\"tokenizer\":\"raw\"},{\"name\":\"app_name\",\"type\":\"text\",\"fast\":true,\"fieldnorms\":false,\"indexed\":true,\"record\":\"basic\",\"stored\":true,\"tokenizer\":\"raw\"},{\"name\":\"message\",\"type\":\"text\",\"fast\":false,\"fieldnorms\":false,\"indexed\":true,\"record\":\"position\",\"stored\":true,\"tokenizer\":\"default\"}],\"tag_fields\":[\"app_name\"],\"mode\":\"lenient\",\"dynamic_mapping\":null}", index_uri: "s3://dev-quickwit-poc/indexes/my_idx" } with Grpc(0.0.0.0:7281)
Jun 30 10:34:20 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:20.041Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
Jun 30 10:34:20 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:20.041Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="my_idx" splits=[SplitIdAndFooterOffsets { split_id: "01G6MW3Q370CCCEE86YQH9C5BM", split_footer_start: 4341838823, split_footer_end: 4342221670 }, SplitIdAndFooterOffsets { split_id: "01G6MX8RM1QE2CVXXW4Y3T83D2", split_footer_start: 3828641765, split_footer_end: 3829057298 }, SplitIdAndFooterOffsets { split_id: "01G6MXS34YNH5AVNHP9MWEH8QS", split_footer_start: 1981272768, split_footer_end: 1981516268 }, SplitIdAndFooterOffsets { split_id: "01G6MY80222E5DV65SQRJJ79JV", split_footer_start: 2538610864, split_footer_end: 2538890359 }, SplitIdAndFooterOffsets { split_id: "01G6MYK20TS8P9QHVTT5MFG4MX", split_footer_start: 2433842269, split_footer_end: 2434101953 }, SplitIdAndFooterOffsets { split_id: "01G6MYY307GCYF5MXEQJCRMTPT", split_footer_start: 2394713447, split_footer_end: 2394971408 }, SplitIdAndFooterOffsets { split_id: "01G6MZ958A8RJM3ZZK4RTX9EAA", split_footer_start: 2371375034, split_footer_end: 2371640066 }, SplitIdAndFooterOffsets { split_id: "01G6MZKV98BYA92DTZPKAECMXA", split_footer_start: 415228357, split_footer_end: 415291041 }, SplitIdAndFooterOffsets { split_id: "01G6MZM63NQ8QR67DB1Q54AKYP", split_footer_start: 2355373354, split_footer_end: 2355627748 }, SplitIdAndFooterOffsets { split_id: "01G6MZQYBDFZ044B64HPK86N8B", split_footer_start: 431290117, split_footer_end: 431356533 }]
Jun 30 10:34:20 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:20.098Z ERROR quickwit_serve::search_api::rest_handler: Error when streaming search results. error=InternalError("Internal error: `Mixed types (u64, i64) for fast field and partition field are not supported.`.")
Jun 30 10:34:20 ip-10-189-129-185 quickwit[21354]: 2022-06-30T10:34:20.101Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed

index config:

version: 0

index_id: my_idx

doc_mapping:
  field_mappings:
    - name: timestamp
      type: i64
      fast: true
    - name: log_level
      type: text
      tokenizer: raw
    - name: app_name
      type: text
      tokenizer: raw
      fast: true
    - name: message
      type: text
      tokenizer: default
      record: position
  tag_fields: [app_name]
  store_source: false
  mode: lenient

indexing_settings:
  timestamp_field: timestamp
  commit_timeout_secs: 60
  docstore_compression_level: 8
  docstore_blocksize: 1000000
  split_num_docs_target: 10000000
  merge_enabled: true
  merge_policy:
    demux_factor: 8
    merge_factor: 10
    max_merge_factor: 12
  resources:
    num_threads: 1
    heap_size: 2000000000

search_settings:
  default_search_fields: [app_name, message]
fulmicoton commented 2 years ago

Ah right. That's why we should avoid using _ as much as possible. https://github.com/quickwit-oss/quickwit/blob/main/quickwit-search/src/search_stream/leaf.rs#L239-L244

The problem is that the fast field is a string. (We introduced string fast fields after this code) The error message is horrible.

Streaming is not supported for string fields. This is something that we can add if it is wanted.

fmassot commented 2 years ago

Ok, I opened a PR to fix at least the error message. @leongshengmin if you need the string field, we can open an issue on it.

fulmicoton commented 2 years ago

@fmassot note that in their use case, app_name is probably low cardinality, so it is actually very reasonable. We might want to be a little bit smart and ship a dictionary with each leaf_search result.

leongshengmin commented 2 years ago

I see, thanks all for the help. Have retried using a timestamp field and am able to get the result without errors.

SELECT count()
FROM url('localhost:8080/api/v1/my_idx/search/stream?query=app_name:my_app&fast_field=timestamp&output_format=click_house_row_binary', RowBinary, 'timestamp Int64')

Query id: e284390c-9c11-4658-83cf-fedfdd0d41fc

┌─count()─┐
│  156459 │
└─────────┘

1 row in set. Elapsed: 1.158 sec. Processed 156.46 thousand rows, 1.25 MB (135.07 thousand rows/s., 1.08 MB/s.)

For our use case, believe non string fields will suffice for now. Will open a new issue if such a requirement does arise.

fmassot commented 2 years ago

Closed by #1691