opensearch-project / ml-commons

ml-commons provides a set of common machine learning algorithms, e.g. k-means, or linear regression, to help developers build ML related features within OpenSearch.
Apache License 2.0
98 stars 136 forks source link

Fine tune error message for model content changed #1630

Closed ylwu-amzn closed 4 months ago

ylwu-amzn commented 1 year ago

When user upload model via URL and didn't provide model hash value, it will throw "model content changed" error, which is hard to understand. We should clearly tell user that model content has value not set.

POST /_plugins/_ml/models/_upload
{
  "name": "all-MiniLM-L6-v2",
  "version": "1.0.0",
  "description": "test model",
  "model_format": "TORCH_SCRIPT",
  "model_config": {
    "model_type": "bert",
    "embedding_dimension": 384,
    "framework_type": "sentence_transformers"
  },
  "url": "https://github.com/opensearch-project/ml-commons/raw/2.x/ml-algorithms/src/test/resources/org/opensearch/ml/engine/algorithms/text_embedding/all-MiniLM-L6-v2_torchscript_sentence-transformer.zip?raw=true"
}

Task

{
  "task_type": "REGISTER_MODEL",
  "function_name": "TEXT_EMBEDDING",
  "state": "FAILED",
  "worker_node": [
    "_QJb--HRS2-7lfq5DCWMiQ"
  ],
  "create_time": 1693505198395,
  "last_update_time": 1693505199947,
  "error": "model content changed",
  "is_async": true
}

Need to calcuate sha256 first

shasum -a 256 all-MiniLM-L6-v2_torchscript_sentence-transformer.zip

Then upload

POST /_plugins/_ml/models/_register
{
    "name": "all-MiniLM-L6-v2",
    "version": "1.0.0",
    "description": "test model",
    "model_format": "TORCH_SCRIPT",
    "model_group_id": "FTNlQ4gBYW0Qyy5ZoxfR",
    "model_content_hash_value": "c15f0d2e62d872be5b5bc6c84d2e0f4921541e29fefbef51d59cc10a8ae30e0f",
    "model_config": {
        "model_type": "bert",
        "embedding_dimension": 384,
        "framework_type": "sentence_transformers"
    },
    "url": "https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/torch_script/sentence-transformers_all-MiniLM-L6-v2-1.0.1-torch_script.zip"
}
TrungBui59 commented 12 months ago

@ylwu-amzn can you assign me to this issue? Thank you

ylwu-amzn commented 12 months ago

Sure, thanks @TrungBui59, assigned to you

ylwu-amzn commented 9 months ago

@TrungBui59 , any update ? We plan to fix this in 2.12, if you don't have enough bandwidth, will ask someone else to help.

TrungBui59 commented 9 months ago

@ylwu-amzn sorry for the delayed response, I was having some personal issues, so I wasn't checking frequently, I am still working on it but it seems like others have posted a PR for this issue