elastic / eland

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

https://eland.readthedocs.io

Apache License 2.0

636 stars 98 forks source link

Check model output when measuring embedding size #535

Closed davidkyle closed 1 year ago

davidkyle commented 1 year ago

532 added code for generating a text_embedding and measuring the number of dimensions to set the `embedding_size` field. The code expected the model output to be a tuple but for some models that is not the case.

embedding_size was added to Elasticsearch in v8.8, the new field is only added if the ES cluster is v8.8 or greater.

Closes #533

davidkyle commented 1 year ago

@sethmlarson I've the test you asked and I've added a check against the version of the Elasticsearch cluster so the script will not use the new feature unless it is supported by Elasticsearch

elastic / eland

Check model output when measuring embedding size #535

532 added code for generating a text_embedding and measuring the number of dimensions to set the embedding_size field. The code expected the model output to be a tuple but for some models that is not the case.

532 added code for generating a text_embedding and measuring the number of dimensions to set the `embedding_size` field. The code expected the model output to be a tuple but for some models that is not the case.