elastic / eland

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
https://eland.readthedocs.io
Apache License 2.0
627 stars 98 forks source link

Fix error finding the model's max input size and add override option #674

Closed davidkyle closed 3 months ago

davidkyle commented 3 months ago

The error in #668 is due to the model's max input length being incorrectly set to a value that is larger than the true max input length. The problem is due to falling back to the max_position_embeddings setting if model_max_length is not found in the config. max_position_embeddings cannot be reliably used to set the max input size as it may be greater than the true max input.

For the situations where the max input length cannot be found in the model's configuration a new --max-model-input-length setting is added to the eland/cli/eland_import_hub_model script allowing the user to manually specify the value. This is an advanced setting and requires the user to knowingly set an accurate value. It is valid to set a smaller value and input will be truncated to that smaller value.

    eland_import_hub_model \
      --cloud-id $ELASTIC_CLOUD_ID \
      --hub-model-id 'sentence-transformers/msmarco-MiniLM-L-12-v3' \
      --task-type text_embedding \
      --max-model-input-length=256 

Closes #668