Open osok opened 1 week ago
Looking through the call trace it looksl ike in
giskard/llm/generators/base.py line 64,
rather than use the supplied model, it uses self.llm_client()
which is set on line 37 self.llm_client = llm_client or get_default_client()
which is openAI
client/init.py line 60
from .openai import OpenAIClient
default_llm_api = get_default_llm_api()
There is some logic for azure or openai.
can we add logic for other openai compliant llm which simply need a base_url set to another server. This would allow a host of other llm servers to participate. I'm using LM Studio in a course for our associates, that I am creating, because students can run that locally, along with quantized model. In the enterprise we would use AWS SageMaker.
"Is it possible to use a local large model to verify the scan function?" some like ollama or HF instance llms function...
I cannot use OpenAI for scan because I get a rate limit error. I have a paid account, but still get a rate limit error. What I really want is to use my local LLM.
Is this something that can be added? or is this tool intentionally only for Azure and OpenAI?
I made the change to my fork. https://github.com/osok/giskard/blob/main/giskard/llm/client/__init__.py
Its about 10 lines of code. Now if the env variable GSK_LLM_BASE_URL is set, it will use that when creating the OpenAI instance.
I made a pull request: https://github.com/Giskard-AI/giskard/pull/1963
Issue Type
Bug
Source
source
Giskard Library Version
2.14.0
Giskard Hub Version
not using
OS Platform and Distribution
Ubuntu 22.04.4 LTS
Python version
Python 3.9.19
Installed python packages
Current Behaviour?
NOTE I added a second comment below which gets to the root of the problem and suggests a fix.
When I call
scan_results = giskard.scan(model=giskard_model)
[cut here for space]
I givwe the full response below, but this sticks out:
https://api.openai.com/v1/chat/completions
I have not configured this anywhere to call to openai, rather the I did configure LM Studio.
Is there a way to globally configure Giskard to usin LM Studio?
Standalone code OR list down the steps to reproduce the issue
I'm using LM Studio Model : TheBloke/Llama 2 13B Q 8.0 GGUF Embeddings : nomic-embeded-text
Here is the code I use to get to this point. I'm using Jyputer Notebook so I'll break it out code / response
response
response
response
then
full results