Sharding Warning

[AbinashSankaran]

When i run a deployed GPT 3.5 model from Azure, i get this warning, "UserWarning: Task supports sharding, but model does not provide context length. Data won't be sharded, prompt might exceed the model's context length. Set context length in your config"

What is the way to set the context_length as i am not able to find properly in the docs anywhere

How to reproduce the behaviour

Config:

[nlp]
lang = "en"
pipeline = ["llm"]
batch_size = 128

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["COMPLIMENT", "INSULT"]

[components.llm.model]
@llm_models = "spacy.Azure.v1"
model_type = "chat"
deployment_name = "gpt-35"
name = "gpt-35"
config = {"temperature": 0.0}
base_url = "https://****.openai.azure.com/"

Code:

from spacy_llm.util import assemble
nlp = assemble("config.cfg")
doc = nlp("You look beautiful!")
print(doc.cats)

Your Environment

• spaCy version: 3.7.4
• Platform: macOS-14.3.1-arm64-arm-64bit
• Python version: 3.9.6

[svlandeg]

Hi! Let me move this to the discussion forum and follow up with you there.