search

elastic / terraform-provider-elasticstack

Terraform provider for Elastic Stack
https://registry.terraform.io/providers/elastic/elasticstack/latest/docs
Apache License 2.0
164 stars 85 forks source link

[Feature] Add Support for ML-Based APIs for Semantic Search #725

Open bhoskins7 opened 3 weeks ago

bhoskins7 commented 3 weeks ago

Is your feature request related to a problem? Please describe. When deploying Semantic search based use cases we require to download the proper model, deploy and possibly test it is available before data can be ingested into the elasticsearch cluster and possibly setup connector to the LLMs / External Models for use at ingest and query time.

Describe the resource you would like to have implemented..

  1. Eland for ability to upload models from hugging face.
  2. Load Model PUT _ml/trained_models/<model_id>
  3. Start Model POST _ml/trained_models/<model_id>/deployment/_start
  4. Get Trained Model GET _ml/trained_models/<model_id>
  5. Create / Update Trained Model Alias PUT _ml/trained_models/<model_id>/model_aliases/<model_alias>
  6. Create Connectors for LLMs or External Embedding Models AWS Bedrock, Azure OpenAI Google Vertex, Cohere, etc.

Describe the solution you'd like Terraform stack functions that can call these apis in preparation of delivering a semantic search use case.

Describe alternatives you've considered Direct / Manual REST API Calls or custom scipts.

jvia commented 2 weeks ago

This would be great! We are starting to experiment with vector search at the moment and having to run eland to managed our models is a bit of a pain. The rest of our cluster is managed with Terraform so having this one outside piece leads to manual cluster work.