search

cortecs-ai / cortecs-py

Lightweight wrapper for cortecs.ai enabling 🔵 instant provisioning
https://cortecs.ai
Apache License 2.0
6 stars 0 forks source link

Instance type is ignored #6

Open alex-cortecs opened 2 weeks ago

alex-cortecs commented 2 weeks ago

When using 

with DedicatedLLM(client=cortecs, model_name='Qwen/Qwen2-7B-Instruct', instance_type='NVIDIA_L40S_1') as llm:
    essay = llm.invoke('Write an essay about dynamic provisioning')
    print(essay.content)

the instance_type is ignored and the recommended instance type is used.

alex-cortecs commented 2 weeks ago

Okay now I understand. I deleted all models in my console and after that choosing instance type works.