Support for all Elasticsearch field types

[masseyke]

There are quite a few field types that have been added to Elasticsearch that are not currently supported in es-hadoop.

Unsupported field types

• [ ] aggregate_metric_double
• [ ] alias
• [ ] annotated_text
• [ ] completion (ignored)
• [ ] constant_keyword
• [x] date_nanos https://github.com/elastic/elasticsearch-hadoop/issues/1653
• [ ] flattened (does not cause a failure on read, but they do not come back)
• [ ] histogram
• [x] ip https://github.com/elastic/elasticsearch-hadoop/issues/1737
• [ ] match_only_text
• [ ] murmur3
• [ ] percolator
• [ ] point
• [ ] range
• [ ] rank_feature
• [ ] rank_features
• [ ] search_as_you_type
• [ ] shape
• [ ] version
• [ ] wildcard https://github.com/elastic/elasticsearch-hadoop/issues/1697
• [ ] dynamic_date_formats https://github.com/elastic/elasticsearch-hadoop/issues/1988
• [ ] dense_vector

I have not tested all of them, but at least some of these cause failures in es-hadoop if you read from an index that uses them.

[rseldner]

Looks like the dense_vector type needs to be added as well. https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html

[sbhatm1213]

Has the dense_vector field type been added ? Or does anyone know of any other way to pull that field on Pyspark ?

[mouhagaye]

I've tested recently with spark reading from index containing most of elastic search field type. Used versions:

• Spark : 3.3.1
• ElasticSearch: 8.12.1
• elasticsearch-spark: 8.12.1

Spark read successfully the index but the these following field type does not appear on the created dataframe:

• constant_keyword
• unsigned_long
• flattened
• integer_range
• float_range
• long_range
• double_range
• date_range
• ip_range
• version
• agg_metric
• histogram
• match_only_text
• completion
• search_as_you_type
• dense_vector
• sparse_vector
• rank_feature
• rank_features
• point
• shape
• percolator