[META] Derived Fields

[qreshi]

Please describe the end goal of this project

"Schema on reads" is a mechanism where query result data can be defined at runtime of the query. This is useful for users who may not be aware of the structure of the data and want to explore its usage before choosing what and how to index it. More simply, this allows users to have additional attributes to query upon post-ingest. Derived fields are an enhancement to OpenSearch that provides this "schema on read" functionality.

Supporting References

The RFC for this feature request can be found here https://github.com/opensearch-project/OpenSearch/issues/1133

Issues

• [x] https://github.com/opensearch-project/OpenSearch/issues/12506
• [x] https://github.com/opensearch-project/OpenSearch/issues/12507
• [x] https://github.com/opensearch-project/OpenSearch/issues/12508
• [x] #14590
• [ ] Scoring support
• [ ] Sorting support

I will update this section with created issues with more details. For the time being I wanted to create the META issue for tracking.

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8] @qreshi Thanks for filing, it would be useful to to create subtasks with additional details for some of those items, looking forward to seeing progress here.

[rishabhmaurya]

List of all tasks associated -

• [x] Derived fields building blocks: DerivedFieldQuery, DerivedFieldScript and DerivedFieldValueFetcher - https://github.com/opensearch-project/OpenSearch/pull/12746
• [x] Derived fields supported field types implementation -boolean, keyword, long, double, ip, geo_point, date - https://github.com/opensearch-project/OpenSearch/pull/12808
• [x] Derived fields Mapper definition and integration through index mapping way - https://github.com/opensearch-project/OpenSearch/pull/12569
• [x] Support for adding multiple values from derived field scripts - https://github.com/opensearch-project/OpenSearch/pull/12837
• [x] Support defining derived field through search requests and rest client. Also adds support for derived fields in fetch and highlight phase - https://github.com/opensearch-project/OpenSearch/pull/12850
• [x] Support for object type in derived fields - https://github.com/opensearch-project/OpenSearch/issues/13143
• • [x] https://github.com/opensearch-project/OpenSearch/pull/13592
• • [x] [DerivedField] object type support in mappings and new settings for derived field #13717
• • [x] https://github.com/opensearch-project/OpenSearch/pull/13720
• • [x] https://github.com/opensearch-project/OpenSearch/pull/13721
• [x] Documentation - https://github.com/opensearch-project/documentation-website/issues/6943
• [ ] Support in dashboards - https://github.com/opensearch-project/OpenSearch-Dashboards/issues/6817
• [ ] Performance benchmark
• [ ] Can it interfere with field level security?

Some additional sub features -

• [ ] Allow removal of derived fields from index mappings
• [ ] Support with concurrent segment search. LeafLookup instance shared among segments within a slice could be problem, assess if its a problem and how to fix it.