elastic / eland

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
https://eland.readthedocs.io
Apache License 2.0
627 stars 98 forks source link

Eland should be Serverless aware when throwing incompatible ES version errors #696

Closed ppf2 closed 1 month ago

ppf2 commented 1 month ago

Currently, eland checks the ES version whek checking compatible.

2024-05-31 12:44:50,385 INFO : Establishing connection to Elasticsearch
2024-05-31 12:44:50,808 INFO : Connected to cluster named '<Serverless Project ID>' (version: 8.11.0)
2024-05-31 12:44:50,808 ERROR : Eland uses PyTorch version 2.1.2 which is incompatible with Elasticsearch versions prior to 8.13. Please upgrade Elasticsearch to at least version 8.13

On Serverless, this is problematic because Serverless does not have the notion of version but it will always return an arbitrary 8.11.0 version number (e.g. in GET /) for backwards compatibility reasons (it does not mean that automatic upgrades on Serverless is not working). So eland will need to be updated to check against build_flavor instead of version.number. Otherwise, Serverless ES projects will always fail with model loading:

"build_flavor": "serverless"