Currently, Flint covering index maps column types from the source table to very basic types in OpenSearch. There is no way to create covering index with other type such as text, IP, and vector etc. This limitation restricts the flexibility and full potential of using Flint with OpenSearch.
What solution would you like?
Flint covering index should automatically create an underlying OpenSearch index with the most suitable types or provide a way for users to configure these types.
Possible solutions include:
Flint registers its own type (UDT in Spark), allowing users to create source tables using it. However, this solution does not help tables created outside Flint extension.
Users can provide type information in the CREATE INDEX statement, e.g., CREATE INDEX all ON test (message TEXT ...), though this is not standard in the SQL world.
Users can achieve this through a materialized view, e.g., CREATE MATERIALIZED VIEW test AS SELECT CAST(message AS TEXT).
What alternatives have you considered?
Users can manually create an OpenSearch index with the desired field types, but this approach is cumbersome and prone to errors if there is a mismatch between the OpenSearch index and the covering index definition.
Do you have any additional context?
Here is the relevant Flint source code that handles data type mappings: FlintDataType.scala#L125.
Is your feature request related to a problem?
Currently, Flint covering index maps column types from the source table to very basic types in OpenSearch. There is no way to create covering index with other type such as
text,IP, andvectoretc. This limitation restricts the flexibility and full potential of using Flint with OpenSearch.What solution would you like?
Flint covering index should automatically create an underlying OpenSearch index with the most suitable types or provide a way for users to configure these types.
Possible solutions include:
CREATE INDEXstatement, e.g.,CREATE INDEX all ON test (message TEXT ...), though this is not standard in the SQL world.CREATE MATERIALIZED VIEW test AS SELECT CAST(message AS TEXT).What alternatives have you considered?
Users can manually create an OpenSearch index with the desired field types, but this approach is cumbersome and prone to errors if there is a mismatch between the OpenSearch index and the covering index definition.
Do you have any additional context?
Here is the relevant Flint source code that handles data type mappings: FlintDataType.scala#L125.