saratvemulapalli
commented
7 months ago Thanks for opening up the issue @ArranDengate-Netapp. I was poking around with the demo[1], I see its using a pre-trained model being downloaded from artifacts.opensearch.org and obviously it just downloads the model but does not have an engine (Eg. pytorch-engine)
I see 3 feature enhancements:
- Can we make ML Commons run without access to network: i.e Have a way to to package and ship dependency libraries a.k.a fat jar. It makes sense, it involves work for packaging, signing these artifacts. We could take this as a feature request.
- Documentation: Document all network access domains for pre-trained models in artifacts.opensearch.org. I think its fair to document, we could do it in the repo[2] or opensearch documentation[3]
- Logging: Improve logging when connection timeout occurs to correctly display the Domain and DNS. We use standard java libraries, we should be able to print the exception stack trace hopefully that has the information we are looking for.
Under this circumstance, GET /_plugins/_ml/models/
tells us the deploy failed, but does not provide a reason. (Not sure if the task API would provide more info - I couldn't see how to get opensearch-py-ml to give me the task ID.)
It should be fairly straight forward to get task ID[4] from deploy model, the deploy model API returns a Task ID which you could query through the Task API[5].
That said, I am fairly new to this repo. I'd like to hear thoughts from other maintainers who are pretty active @ylwu-amzn @austintlee @HenryL27 .
[1] https://opensearch-project.github.io/opensearch-py-ml/examples/demo_ml_commons_integration.html [2] https://github.com/opensearch-project/ml-commons/tree/main/docs [3] https://opensearch.org/docs/latest/ml-commons-plugin/ [4] https://opensearch-project.github.io/opensearch-py-ml/examples/demo_ml_commons_integration.html#Step-2:-Load-Model [5] https://opensearch.org/docs/latest/ml-commons-plugin/api/tasks-apis/index/
ylwu-amzn
ArranDengate-Netapp
brunowcs
manzke
For clusters in a corporate setting, internet access is often restricted with an egress firewall.
However, the ML commons plugin needs internet access to download dependencies, even when using a local model.
It would be good to improve the user experience in this situation. Some ideas:
I see this behaviour when using the
all-MiniLM-L12-v2model locally on OpenSearch 2.11.1, using the TorchScript model file and config from the list of pre-trained models, deploying from a local zip file with the steps fromopensearch-py-ml's demo notebook. I have made some suggestions based on my experience below, but I'm not sure if the ONNX model would have different dependencies than the TorchScript model, or if other models have different dependencies (eg, whetherall-mpnet-base-v2is going to have different dependencies thanall-MiniLM-L12-v2).Packaging
When using a local Torch model on a server with restricted internet access, deploying the model fails if the server cannot access
publish.djl.ai. In ml-commons code, this URL is mentioned by thepytorch-enginelibrary.It might be possible to package a fat jar with dependencies to avoid this issue? This was previously discussed in the OpenSearch forums.
Documentation
It would be useful to document:
Currently, the plugin appears to need network access to the following URLs when deploying, even when using a local model:
[WARN ][a.d.h.z.HfModelZoo ] [ip-172-31-58-14.ec2.internal] Failed to download Huggingface model zoo index: NLP.FILL_MASK; not sure if this has consequences later)Logging
Another way to improve this experience would be to log more information when there is a failure downloading dependencies.
When deploying a local model, if an egress firewall is configured to drop packets to destinations that are not explicitly permitted, we get an error that doesn't tell us which destination we were trying to reach - from this, it is not obvious what address needs to be whitelisted. Here are the OpenSearch logs when deploying a local model under these circumstances:
Under this circumstance, GET
/_plugins/_ml/models/<model-id>tells us the deploy failed, but does not provide a reason. (Not sure if the task API would provide more info - I couldn't see how to get opensearch-py-ml to give me the task ID.)Please note, the above is assuming that DNS is permitted. If the egress firewall is also preventing DNS, the error is more useful and does contain the domain that needs to be whitelisted: