Open jkfnc opened 1 year ago
@jkfnc
AzureOpenAI
in guidance/llms/_azure_openai.py
uses msal
. It seems to need the public enterprise aplication PublicClientApplication
, which is one of the Azure service, and seems to be unclear to deal with Azure OpenAI Service in msal.PublicClientApplication
in a guidance/llms/_azure_openai.py
's manner.
Thus, I've temporarily fixed by overwriting guidance/llms/_openai.py
as follows.
class OpenAI(LLM):
cache = LLM._open_cache("_openai.diskcache")
- def __init__(self, model=None, caching=True, max_retries=5, max_calls_per_min=60, token=None, endpoint=None,
+ def __init__(self, model=None, deployment_id=None, caching=True, max_retries=5, max_calls_per_min=60, token=None, endpoint=None,
temperature=0.0, chat_mode="auto", organization=None, allowed_special_tokens={"<|endoftext|>", "<|endofprompt|>"}):
+ self.deployment_id = deployment_id
...
def _library_call(self, **kwargs):
""" Call the OpenAI API using the python package.
Note that is uses the local auth token, and does not rely on the openai one.
"""
prev_key = openai.api_key
prev_org = openai.organization
assert self.token is not None, "You must provide an OpenAI API key to use the OpenAI LLM. Either pass it in the constructor, set the OPENAI_API_KEY environment variable, or create the file ~/.openai_api_key with your key in it."
openai.api_key = self.token
openai.organization = self.organization
+ if self.deployment_id is not None:
+ openai.api_base = os.environ.get("OPENAI_API_BASE", None)
+ openai.api_type = "azure"
+ openai.api_version = os.environ.get("OPENAI_API_VERSION", None)
if self.chat_mode:
kwargs['messages'] = prompt_to_messages(kwargs['prompt'])
del kwargs['prompt']
del kwargs['echo']
del kwargs['logprobs']
# print(kwargs)
- out = openai.ChatCompletion.create(**kwargs)
+ out = openai.ChatCompletion.create(deployment_id=self.deployment_id, **kwargs)
out = add_text_to_chat_mode(out)
else:
- out = openai.Completion.create(**kwargs)
+ out = openai.Completion.create(deployment_id=self.deployment_id, **kwargs)
openai.api_key = prev_key
openai.organization = prev_org
return out
Then, you can use azure openai like:
gpt4 = guidance.llms.OpenAI(
model="gpt-4",
deployment_id=<your_deployment_id>
)
If you don't want to bother touching inside the library, you can also do this monkeypatch:
def library_call_with_engine(**kwargs):
kwargs['engine'] = "text-davinci-003"
return guidance.llm._library_call(**kwargs)
# set the default language model used to execute guidance programs
guidance.llm = guidance.llms.OpenAI("text-davinci-003")
guidance.llm.caller = library_call_with_engine
also note that if you have an OPENAI_API_KEY env var set and in your code type:
openai.api_key = os.getenv("AZURE_OPENAI_API_KEY")
The guidance.llms.OpenAI() constructor will load the OPEN_API_KEY env var and try to use that. You'll get an invalid authorization error due to that.
Make sure NOT to set the openai.api_key in your code ... INSTEAD pass your AZURE_OPENAI_API_KEY to the constructor like this:
self.guidance_llm = guidance.llms.OpenAI(self.gptmodel.value, token=os.getenv("AZURE_OPENAI_API_KEY"))
It's due to this code in the OpenAI constructor:
# fill in default API key value
if token is None: # get from environment variable
token = os.environ.get("OPENAI_API_KEY", getattr(openai, "api_key", None))
if token is not None and not token.startswith("sk-") and os.path.exists(os.path.expanduser(token)): # get from file
print("Loading token from file.")
with open(os.path.expanduser(token), 'r') as file:
token = file.read().replace('\n', '')
if token is None: # get from default file location
try:
with open(os.path.expanduser('~/.openai_api_key'), 'r') as file:
token = file.read().replace('\n', '')
except:
pass
Hi guys,
Is anyone working on trying to solve this?
I used the monkey patch method posted here and works fine for now
Jack Neil, MD CEO | Hank AI, Inc. 711 East Main Street, Suite E, Lexington, SC 29072 @.*** (w) +1.904.GET.HANK
(m) +1.803.466.7551
Personal: LinkedIn ( https://www.linkedin.com/in/jackneil/ ) | Twitter ( https://twitter.com/realjackneil ) | Github ( https://github.com/jackneil ) | Dot ( https://dot.cards/jackmd ) | Reddit ( https://www.reddit.com/user/jackusc ) | Discord ( https://discordapp.com/users/318125799456768001 ) HANK: Website ( https://hank.ai ) | LinkedIn ( https://www.linkedin.com/company/hankai/ ) | Twitter ( https://twitter.com/LETHANKHELP ) | Github ( https://github.com/hankai ) | Youtube ( @.*** )
Sent via Superhuman iOS ( @.*** )
On Tue, May 30 2023 at 5:07 AM, William Brach < @.*** > wrote:
Hi guys,
Is anyone working on trying to solve this?
— Reply to this email directly, view it on GitHub ( https://github.com/microsoft/guidance/issues/87#issuecomment-1568064431 ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AGFRM5SEJZNXDZ6CW3FK5IDXIW2GTANCNFSM6AAAAAAYJVM5YM ). You are receiving this because you commented. Message ID: <microsoft/guidance/issues/87/1568064431 @ github. com>
@jackneil
Could you please share how to do more parameter passing? Like AZURE_OPEN_AI_API_KEY and endpoint url?
I am getting - InvalidRequestError: Resource not found
I’ll post the code used and quick how to later today when I get back to work computer
Here ya go. Hopefully this helps Get your endpoint url from your azure service in portal.azure.com Get your openai api key from your azure service in portal.azure.com
Some of these functions were inside a class, thus the 'self' in the functions and some variables. you'll need to edit those
class GPTModel(Enum):
GPT4 = "gpt-4"
GPT3_5 = "gpt-3.5-turbo"
GPT3 = "text-davinci-003"
class GPTModel_Azure(Enum):
GPT3_5 = "gpt_35_20230527" #"gpt-35-turbo" #USE YOUR DEPLOYMENT ID
GPT4 = "gpt-4"
GPT3 = "text-davinci-003"
gptmodel_to_gptmodelazure = {
GPTModel.GPT4: GPTModel_Azure.GPT4,
GPTModel.GPT3_5: GPTModel_Azure.GPT3_5
}
#monkeypatch for using azure's openai api endpoints. they expect a deployment_id and engine parameter
def library_call_with_engine(self, **kwargs):
kwargs['engine'] = gptmodel_to_gptmodelazure[self.gptmodel].value #have to map the GPTModel (model names openai uses) to the GPTModel_Azure (model name=deploymentid azure openai models use) before getting the text value
kwargs['deployment_id'] = gptmodel_to_gptmodelazure[self.gptmodel].value
print("Library call ...")
return self.guidance_llm._library_call(**kwargs)
if useAzure:
openai_api_key = os.getenv("AZURE_OPENAI_API_KEY")
openai_api_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
self.guidance_llm = guidance.llms.OpenAI(self.gptmodel.value, token=openai_api_key)
self.guidance_llm.caller = self.library_call_with_engine
openai.api_type = "azure"
openai.api_base = openai_api_endpoint #like: https://openai-medicalcoder.openai.azure.com/ or https://hank-openai-europe.openai.azure.com/
openai.api_version = "2023-05-15" #3-15-preview"
else:
if openai_api_key is None: openai_api_key = os.getenv("OPENAI_API_KEY")
self.guidance_llm = guidance.llms.OpenAI(gptmodel.value) #load up for gpt4, 3, etc
openai.api_key = openai_api_key
openai.api_base = "https://api.openai.com/v1"
Then use it something like this:
experts = guidance(self.guidance_handlebars_ourformat, llm=self.guidance_llm)
response = experts(doctype=doctype.value)
Hi all! Sorry this took a while to address. Can you let me know if the following example works when you use 0.0.61? Thanks!
import guidance
llm = guidance.llms.OpenAI(
'text-davinci-003',
api_type='azure',
api_key='sk-...',
api_base='https://example-endpoint.openai.azure.com',
api_version='2023-05-15',
deployment_id='...',
caching=False
)
program = guidance("""My favorite flavor is{{gen 'flavor' max_tokens=10 stop="."}}""", llm=llm)
program()
@slundberg thanks for fix!
it works.
deployment_id == model deployment name at https://oai.azure.com/
I am still not able to get the deployment id, can anyone help?
To Reproduce Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.
System info (please complete the following information):
guidance.__version__
): guidance 0.0.53Error: Traceback (most recent call last): File "c:\Users\ABCDEF\NewSource\guidance_test.py", line 10, in
gpt3 = guidance.llms.OpenAI(model="deployment_name",endpoint=os.environ["AZURE_OPENAI_ENDPOINT"])
File "C:\Users\ABCDEF\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py", line 113, in init
self._tokenizer = tiktoken.get_encoding(tiktoken.encoding_for_model(model).name)
File "C:\Users\ABCDEF\AppData\Roaming\Python\Python310\site-packages\tiktoken\model.py", line 71, in encoding_for_model
raise KeyError(
KeyError: 'Could not automatically map deployment_name to a tokeniser. Please use
tiktok.get_encoding
to explicitly get the tokeniser you expect.'Even if we add mapping for cl100k_base in tiktoken, it gives this 404 error.