zhichao-aws
commented
8 months ago Different processors may have different pre-processing or post-processing logics. Can we customize these in the new processor? For example, when there are multiple inference fields in one document, sparse_encoding processors will obtain different response as follows:
# case remote inference
[{
"response":{
[
{ TOKEN_WEIGHT_MAP},
{ TOKEN_WEIGHT_MAP}
]
}
}]
# case local deploy
[{
"response":{
[
{ TOKEN_WEIGHT_MAP}
]
}
},{
"response":{
[
{ TOKEN_WEIGHT_MAP}
]
}
}]The processor should fetch those TOKEN_WEIGHT_MAP and assign them to the target field. Can you give an example of how to configure a ml_inference processor with customized logics? Thanks!
mingshl
zane-neo
HenryL27
chishui
xinyual
austintlee
br3no
ylwu-amzn
Problem statement
Currently, there are different implementations of search processors/ingest processors that use a machine learning model, e.g, TextEmbeddingProcessor for text-embedding models , GenerativeQAResponseProcessor for large language models, PersonalizeRankingResponseProcessor for reranking models housing in AWS personalized Service. Looking forward, when each type of machine learning models has a separate type of processor, the number of processors will grow to be enormous. It will be in-convenient for users when configuring different processors. However, ML Commons plugin supports connecting to a foundation model hosted on an external platform and uploading your own pre-trained model to the OpenSearch cluster, users can utilize the model_id from ml-commons plugins to apply in search/ingest process. We can simplified the multiple implementations of search processors/ingest processors that use a machine learning model.
Motivation:
To Improve the ease of using machine learning models to process ingest/search requests, we are introducing a Machine Learning Inference Processor to OpenSearch-ml-common to uses a machine learning model to read from the data and add the prediction outcome to the data that is being ingested through the ingest pipeline, or return the prediction outcomes along with the search response that is returned through the search pipeline.
Scope:
Out of Scope:
-ML inference Processor is focusing on model inferences and does not handle data transformations. Developers would consider data formatting methods before using ML Inference Processors if the documents and search hit does not match the format of model input. For example, adding the preprocess methods to model connectors (example), utilizing data transforming processing (e.g Split Processors, JsonProcessor ).
Proposed Design:
Create ML Inference processors(ingest side), ML Inference search requests processors, ML Inference search response processors that share the same parameters and extend from same interface that handles getModelInferenceResult.
ML Inference Processors parameters:
truefor local models andfalsefor externally hosted modelsSample Process:
using the following example for a text embedding remote model wupL7Y0Bm1mYgYg_PasK that is connected in ml-common,
Added after gathering feedbacks for different use cases,
0. using multiple rounds of predictions
Sometimes, a model only accept one model input fields, and we would like to predicts on multiple fields, we need to run the model multiple times. The inference processors can run one model with multiple inference.
in this setting, it will run the model twice and mapping the output accordingly to two document fields.
the sample response would be
Handling object type model input:
for example, I want to use a language classification model , this model's predict function is expecting an object, in the format of {"input": ["text"]}, this can be a good complicated case, it's object with a map of list.
1. Using the one field as ml input
then in the ml connector, if the use case is using the one field from the document to identify we can define the connectors and in the response body
in this request body, it helps formatting the parameters.input field into the desire model input format. In using this model to predict in ml common, we don't need to worry about the format of an object of a map with list, instead, we use parameters.input as we defined in the connectors.
Let's use the inference processors during ingestions for
reviewsfield:Now the documents getting ingest already has the model output field named
reviews_language_classificationSince I turned on dynamic mapping, to use the language label field, we can search using doc path
reviews_language_classification.labelin search queries .2. Using the multiple fields as ml input
Another use case, @zhichao-aws also mentioned that many models now accepts multiple model input fields, nowadays, text_embedding models and classification models accepts multiple model inputs, we just need to config the connectors properly to meet this multiple input fields requirements. Similarly in the response body, we config two input fields.
In this case, it's looking for two input fields to the document, and format properly, the ml_inference processor will handle the mappings for reviews -> input1, products -> input2.
3. Formatting with other processors.
in the connectors, it's supported writing post_process_function, and also we can use other processors before ml_inference_processor to handle model input field format or after to handle model output format.
I am continuing the for the second step, and would like parse the model output field
reviews_products_language_classificationinto two fields. this is to seperate an array and append to new fields, we can use a script processor and remove processor to bundle with it. In the future, we can also add a new type of processor maybe called "seperate_append" processor then it would be easier to use.let's modify the ingest pipeline for the same index
then when ingesting the same document, it returns