Open tanujjain opened 11 months ago
Hey, I found a workaround for this. It can be solved using LangChain in the Python API. You can define your LLM in LangChain using the Azure Chat OpenAI object and pass that as an argument to the LLMRails class.
For example,
from langchain.chat_models import AzureChatOpenAI
llm = AzureChatOpenAI(<your-parameters-here>)
from nemoguardrails import LLMRails, RailsConfig
config = RailsConfig.from_path("path/to/config")
app = LLMRails(config, llm=llm)
new_message = app.generate(messages=[{
"role": "user",
"content": "Hello! What can you do for me?"
}])
An additional error I was noticing here is that app.generate kept telling me that I can't run synchronous calls inside async code, not really sure why. I got it running by replacing app.generate with app.generate_async but that's probably not the best way to solve this problem.
@tanujjain @chilicrabcakes facing a similar issue while trying to work with guardrails . Parameter temperature does not exist for NoneType Error 'NoneType' object has no attribute 'agenerate_prompt' while execution check_jailbreak Traceback (most recent call last): File "C:\Python311\Lib\site-packages\nemoguardrails\actions\action_dispatcher.py", line 125, in execute_action result = await fn(**params) ^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\nemoguardrails\actions\jailbreak_check.py", line 50, in check_jailbreak check = await llm_call(llm, prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\nemoguardrails\actions\llm\utils.py", line 31, in llm_call result = await llm.agenerate_prompt(
Any suggestions ,have you faced this issue
@ansumanparija007 : looks like llm
is set to None
in your case. Can you provide more details on the config?
Hi @drazvan i am try in to use the following config to connect to azure open ai- chat_model = AzureChatOpenAI( openai_api_type="azure", openai_api_version="2023-03-15-preview", openai_api_key=azure_openai_key, deployment_name=azure_openai_model, openai_api_base=azure_openai_endpoint ) app = LLMRails(config=config, llm=chat_model)
new_message = app.generate(messages=[{ "role": "user", "content": "What's the latest fashion trend?" }])
i hope currently guradrails supports azure open ai ,pls let me know
any luck with this @drazvan ?
I can't test this directly as I don't have an Azure Key. Can you confirm the chat_model
instance works correctly? i.e. completion = chat_model("some text")
. And if it does, can you share the complete error stack trace? Thanks.
Yeah chat_model works fine .but while working with guardrails following error is thrown - Parameter temperature does not exist for NoneType Error 'NoneType' object has no attribute 'agenerate_prompt' while execution check_jailbreak Traceback (most recent call last): File "C:\Users\testcopilot2\guardrails\chatbot-guardrails\env\Lib\site-packages\nemoguardrails\actions\action_dispatcher.py", line 125, in execute_action result = await fn(**params) ^^^^^^^^^^^^^^^^^^ File "C:\Users\testcopilot2\guardrails\chatbot-guardrails\env\Lib\site-packages\nemoguardrails\actions\jailbreak_check.py", line 50, in check_jailbreak check = await llm_call(llm, prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\testcopilot2\guardrails\chatbot-guardrails\env\Lib\site-packages\nemoguardrails\actions\llm\utils.py", line 31, in llm_call result = await llm.agenerate_prompt( ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'agenerate_prompt' new_message: {'role': 'assistant', 'content': "I'm sorry, an internal error has occurred."}
Ok, I think I'm starting to understand the issue. Can't test, so bear with me. Can you try adding the following:
app = LLMRails(config=config, llm=chat_model)
app.runtime.register_action_param("llm", chat_model)
thanks @drazvan this error is gone.ll try to play with it and let you know ..
Hi @drazvan while running hallucination cheack it says Hallucination rail can only be used with OpenAI LLM engines.Current LLM engine is AzureChatOpenAI. Any ETA when it will be supported for other LLM models?
Hi @drazvan @ansumanparija007 @tanujjain I believe we can help with this issue. I’m the maintainer of LiteLLM https://github.com/BerriAI/litellm
TLDR:
We allow you to use any LLM as a drop in replacement for gpt-3.5-turbo
.
If you don't have access to the LLM you can use the LiteLLM proxy to make requests to the LLM
You can use LiteLLM in the following ways:
This calls the provider API directly
from litellm import completion
import os
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-key" #
os.environ["COHERE_API_KEY"] = "your-key" #
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion(model="command-nightly", messages=messages)
this is great if you don’t have access to claude but want to use the open source LiteLLM proxy to access claude
from litellm import completion
import os
## set ENV variables
os.environ["OPENAI_API_KEY"] = "sk-litellm-5b46387675a944d2" # [OPTIONAL] replace with your openai key
os.environ["COHERE_API_KEY"] = "sk-litellm-5b46387675a944d2" # [OPTIONAL] replace with your cohere key
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion(model="command-nightly", messages=messages)
I've been trying to use Nemo-Guardrails with Azure but keep getting the dreaded
"openai.error.InvalidRequestError: Resource not found"
error. Outside of Nemo-Guardrails, my Azure credentials work withopenai python api
as well as langchain Azure apis (bothAzureOpenAI and AzureChatOpenAI
).I tried the helloworld example and ran the cli command:
nemoguardrails chat --config=config/hello_world
. Also tried running the basic usage example for python api. Have tried setting engine parameter in config.yml toopenai
andazure
.Checking the engine parameter in the config from the documentation-
engine: the LLM provider; currently, only "openai" is supported.
But also from the documentation-
You can use any LLM provider that is supported by LangChain, e.g., ai21, aleph_alpha, anthropic, anyscale, azure ..
So, is Azure supported?
My specifics-
Python 3.10.11
nemoguardrails==0.3.0
openai==0.27.7
langchain==0.0.167
conda 22.9.0
macOS Ventura(M1 Max chip)