Request for Custom Model Translator Support

[Quincynickyoung]

Description

Hello DJL team,

I am currently working on deploying a custom text vectorization model using DJL Serving. My model is based on PyTorch and utilizes BERT for text vectorization. However, I have encountered an issue: there is no suitable translator available in the existing DJL Serving framework that fits the specific needs of my model.

Model Details Model Type: Text Vectorization Framework: PyTorch Description: The model takes a string of text as input and outputs a vector representation.

Current Challenge The existing translators in DJL, such as those for object detection, question answering, sentiment analysis, and text generation, do not align with the requirements of a text vectorization model. My model requires a custom translator to handle the conversion of text inputs into the model's expected format and to process the model's vector output.

Suggested Solution It would be highly beneficial for DJL Serving to provide support for custom model translators that users can define according to their specific model requirements. This feature would greatly enhance the flexibility of DJL Serving, making it more accommodating for a variety of custom models.

Additional Context I have attempted to implement a custom translator in Java following the DJL guidelines but found the process challenging due to my limited experience with Java programming. A more accessible way to define custom translators, possibly through configuration or simpler programming methods, would be extremely helpful.

Thank you for considering this request. I am looking forward to any suggestions or guidance you could provide on this matter.

Best regards, quincy

[frankfliu]

You can use our built-in TextEmbeddingTranslatorFactory. We have many popular Huggingface TextEmbedding model in our model zoo already, you can use them out of the box, You can create a serving.properties file:

engine=PyTorch
option.model_id=djl://ai.djl.huggingface.pytorch/sentence-transformers/all-MiniLM-L6-v2

or even simple pass the url in command line:

djl-serving -m djl://ai.djl.huggingface.pytorch/sentence-transformers/all-MiniLM-L6-v2

[Quincynickyoung]

You can use our built-in TextEmbeddingTranslatorFactory. We have many popular Huggingface TextEmbedding model in our model zoo already, you can use them out of the box, You can create a serving.properties file:

engine=PyTorch
option.model_id=djl://ai.djl.huggingface.pytorch/sentence-transformers/all-MiniLM-L6-v2

or even simple pass the url in command line:

djl-serving -m djl://ai.djl.huggingface.pytorch/sentence-transformers/all-MiniLM-L6-v2

Hello Frank,

First of all, I'd like to express my gratitude for your prompt and helpful response! The information about the built-in TextEmbeddingTranslatorFactory and the Huggingface TextEmbedding models in the DJL model zoo is very valuable to me.

Regarding your suggestions, I have a couple of follow-up questions for which I would appreciate further guidance:

List of Models in the Model Zoo: Could you please direct me on where to find a list of all available Huggingface TextEmbedding models in the DJL model zoo? I am interested in exploring the different model options to find the one that best fits my requirements.

Using Models in Docker: I am currently deploying DJL Serving in a Docker container. I would like to know if there is a way to pre-download models from the model zoo locally and load them directly in the Docker container, instead of downloading them online. This would be crucial for improving model load times and handling potential network issues.

Thank you once again for your assistance and time. I look forward to your reply to better utilize DJL and its resources.

[frankfliu]

List of Models in the Model Zoo

There are three ways to check the model list:

1. run ./gradlew listModels in djl repo, see: https://docs.djl.ai/docs/load_model.html#list-available-models-using-djl-command-line
2. Use `ModelZoo.listModels()' API
3. Download index files from DJL model zoo:

curl -O https://mlrepo.djl.ai/model/nlp/text_embedding/ai/djl/huggingface/pytorch/models.json.gz
curl -O https://mlrepo.djl.ai/model/nlp/token_classification/ai/djl/huggingface/pytorch/models.json.gz
...

Download model artifacts from model zoo

curl -O https://mlrepo.djl.ai/model/nlp/text_embedding/ai/djl/huggingface/pytorch/sentence-transformers/all-MiniLM-L12-v2/0.0.1/all-MiniLM-L12-v2.zip

Import models from Huggingface You can use model_zoo_importer.py to import model to DJL model zoo format.

[Quincynickyoung]

List of Models in the Model Zoo

There are three ways to check the model list:

1. run ./gradlew listModels in djl repo, see: https://docs.djl.ai/docs/load_model.html#list-available-models-using-djl-command-line
2. Use `ModelZoo.listModels()' API
3. Download index files from DJL model zoo:

curl -O https://mlrepo.djl.ai/model/nlp/text_embedding/ai/djl/huggingface/pytorch/models.json.gz
curl -O https://mlrepo.djl.ai/model/nlp/token_classification/ai/djl/huggingface/pytorch/models.json.gz
...

Download model artifacts from model zoo

curl -O https://mlrepo.djl.ai/model/nlp/text_embedding/ai/djl/huggingface/pytorch/sentence-transformers/all-MiniLM-L12-v2/0.0.1/all-MiniLM-L12-v2.zip

Import models from Huggingface You can use model_zoo_importer.py to import model to DJL model zoo format.

Thanks a lot!