docs: using the model with sagemaker

[greenpau]

Hi,

I am following this guide to deploy instructor-embedding on Amazon SageMaker.

https://www.philschmid.de/custom-inference-huggingface-sagemaker

I've created model.tar.gz that contains cached version of the model.

drwxr-xr-x root/root         0 2023-06-20 15:40 model/
-rw-r--r-- root/root      1477 2023-06-20 15:33 model/.gitattributes
drwxr-xr-x root/root         0 2023-06-20 15:40 model/.ipynb_checkpoints/
-rw-r--r-- root/root     66318 2023-06-20 15:33 model/.ipynb_checkpoints/README-checkpoint.md
-rw-r--r-- root/root       122 2023-06-20 15:33 model/.ipynb_checkpoints/config_sentence_transformers-checkpoint.json
drwxr-xr-x root/root         0 2023-06-20 15:33 model/1_Pooling/
-rw-r--r-- root/root       270 2023-06-20 15:33 model/1_Pooling/config.json
drwxr-xr-x root/root         0 2023-06-20 15:33 model/2_Dense/
-rw-r--r-- root/root       116 2023-06-20 15:33 model/2_Dense/config.json
-rw-r--r-- root/root   3146603 2023-06-20 15:33 model/2_Dense/pytorch_model.bin
-rw-r--r-- root/root     66318 2023-06-20 15:33 model/README.md
-rw-r--r-- root/root      1529 2023-06-20 15:33 model/config.json
-rw-r--r-- root/root       122 2023-06-20 15:33 model/config_sentence_transformers.json
-rw-r--r-- root/root       461 2023-06-20 15:33 model/modules.json
-rw-r--r-- root/root 1339823867 2023-06-20 15:33 model/pytorch_model.bin
-rw-r--r-- root/root         53 2023-06-20 15:33 model/sentence_bert_config.json
-rw-r--r-- root/root       2201 2023-06-20 15:33 model/special_tokens_map.json
-rw-r--r-- root/root     791656 2023-06-20 15:33 model/spiece.model
-rw-r--r-- root/root    2422360 2023-06-20 15:33 model/tokenizer.json
-rw-r--r-- root/root       2407 2023-06-20 15:33 model/tokenizer_config.json
-rw-r--r-- root/root       2177 2023-06-22 18:17 code/inference.py
-rw-r--r-- root/root         70 2023-06-22 18:17 code/requirements.txt

In the inference.py I load the model from the model directory in model.tar.gz.

from transformers import AutoTokenizer, AutoModel

    tokenizer = AutoTokenizer.from_pretrained(model_dir + "/model")
    model = AutoModel.from_pretrained(model_dir + "/model")

The model type for this one comes up as T5Model and it does not have encode method.

[INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: 'T5Model' object has no attribute 'encode' : 400",

Which method and syntax do I use to perform the embedding?

[greenpau]

@hongjin-su , any advice here?

[greenpau]

The types are:

• model: transformers.models.t5.modeling_t5.T5Model
• tokenizer: transformers.models.t5.tokenization_t5_fast.T5TokenizerFast

[greenpau]

Tried using tokenizer/model.

            encoding = tokenizer(pair["instruction"], pair["text"], return_tensors="pt")            
            input_ids = encoding["input_ids"]
            attention_mask = encoding["attention_mask"]
            # output is transformers.modeling_outputs.BaseModelOutputWithPastAndCrossAttentions
            output = model.encoder(input_ids=input_ids, attention_mask=attention_mask, return_dict=True)
            # The shape of the last hidden state is torch.Size([1, 16, 1024])
            output_last_hidden_state = output.last_hidden_state

The last_hidden_state is [1, 16, 1024]. How do I get 768-dimensional encoding out of it?

[hongjin-su]

Hi, Thanks a lot for your interests in the INSTRCUTOR model!

You may try to load the model via:

from InstructorEmbedding import INSTRUCTOR
model = INSTRUCTOR('hkunlp/instructor-large')

[hongjin-su]

Please re-open the issue if you have any questions or comments!