search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.75k stars 1.81k forks source link

[BUG] OpenSearch should create _seq_no field mapping at Index creation itself and NOT after atleast 1 doc is ingested #16185

Open eirsep opened 1 month ago

eirsep commented 1 month ago

Describe the bug

__seq_no_ field is not created by OpenSearch at index creation time. It seems to be created at the time of ingestion of 1 st document for that index

When our plugin (Security Analytics) fetches data from an index, it tries to sort by _seq_no which fails on empty index instead of returning empty result.

Related component

Indexing

To Reproduce

  1. Create index
  2. Before ingesting data, search on index with sort field as _seq_no 3.Error shows up 
    
Caused by: org.opensearch.index.query.QueryShardException: No mapping found for [_seq_no] in order to sort on
    at org.opensearch.search.sort.FieldSortBuilder.resolveUnmappedType(FieldSortBuilder.java:564) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:411) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.search.sort.SortBuilder.buildSort(SortBuilder.java:168) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.search.SearchService.parseSource(SearchService.java:1268) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.search.SearchService.createContext(SearchService.java:998) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:606) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:579) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1583) ~[?:?]

Expected behavior

OpenSearch should use a protected index template that defines metadata mappings like _version, _seq_no for all indices at index createion time itself

Additional Details

No response

bharath-techie commented 3 weeks ago

When our plugin (Security Analytics) fetches data from an index, it tries to sort by _seq_no which fails on empty index instead of returning empty result.

@eirsep can't this be handled in plugin ? any specific use case on why we do this query ?

bharath-techie commented 3 weeks ago

[Triage attendees - 1 2 3 4]