search

opensearch-project / ml-commons

ml-commons provides a set of common machine learning algorithms, e.g. k-means, or linear regression, to help developers build ML related features within OpenSearch.
Apache License 2.0
91 stars 129 forks source link

[META] Support local model in ML inference processor #2499

Closed ylwu-amzn closed 3 months ago

ylwu-amzn commented 4 months ago

We have released ML inference processor in 2.14 which only supports remote model. We are going to enhance it to support local model too.

asfoorial commented 3 months ago

Does this mean models deployed inside OpenSearch ml-common?

dhrubo-os commented 3 months ago

Does this mean models deployed inside OpenSearch ml-common?

Yes, it is going to be released with 2.15. Related PR: https://github.com/opensearch-project/ml-commons/pull/2508

asfoorial commented 3 months ago

Great! Now is it possible to deploy a small language model such as microsoft/Phi-3-mini-4k-instruct-onnx on OpenSearch and use this feature to do question answering inferencing within OpenSearch? If yes, then would you please share the steps?

ylwu-amzn commented 3 months ago

@asfoorial , no, this feature is to support using current supported local models in ML inference processor. SLM is not a supported local model type. So you can't deploy it inside OpenSearch cluster. You can deploy it somewhere else and use connector to connect. One update in 2.15 is we support private IP setting https://github.com/opensearch-project/ml-commons/pull/2534