Is it possible to evaluate using models from Azure OpenAI Service?

[ayrtondenner]

⁉️ Discussion/Question

I have a model currently available via Azure OpenAI, is it possible to run a prompt through it passing AZURE_OPENAI_API_KEY instead of OPENAI_API_KEY for instance?

[NivekT]

Hi @ayrtondenner, that definitely is a use case that we'd like to support.

I will look into it, but if someone wants to try implementing this, feel free to comment here. Two things I think we will need:

• [x] Support for Azure OpenAI in code/notebook
• [ ] Support for Azure OpenAI in Playground

Just curious - is your primary use case within in the playground or within a notebook?

[ayrtondenner]

Hi @NivekT , thanks for the answer.

In our case, we are looking to use prompttools via code, since our wish is to integrate it with an automatic pipeline to test our prompts before putting into production. So the idea here is to test a new prompt and check if it will break any known chat case.

[NivekT]

@ayrtondenner We have updated our OpenAICompletionExperiment API to support Azure. Here is a code example.

Let us know how it is working for you. Happy to make further changes if you experience any issue. Feel free to open a PR to make changes as well.

I will follow up and update the ChatExperiment as well. If there is interest to support Azure within the playground, please comment and let us know.

[NivekT]

@ayrtondenner I can confirm that we have added support for Azure in both chat and text completion. Let me know if it is working for you. We would also love to chat about your use cases.

[ayrtondenner]

Is that already added in the pip library? I just tried it in a new Colab notebook but it didn't work:

TypeError: OpenAICompletionExperiment.__init__() got an unexpected keyword argument 'azure_openai_service_configs'

[NivekT]

Hi @ayrtondenner, we have not pushed a new package but we will do that today. In the meantime, it is available if you install from source by cloning the GitHub repo.

[NivekT]

@ayrtondenner We have pushed a new package. Let us know if it works for you!

[ayrtondenner]

I got the following new error:

from prompttools import utils

AttributeError                            Traceback (most recent call last)
[<ipython-input-4-aab89cdcb7ec>](https://localhost:8080/#) in <cell line: 4>()
      2 from prompttools.experiment import OpenAIChatExperiment
      3 
----> 4 from prompttools import utils

9 frames
[/usr/local/lib/python3.10/dist-packages/numpy/__init__.py](https://localhost:8080/#) in __getattr__(attr)
    309             return val
    310 
--> 311         if attr in __future_scalars__:
    312             # And future warnings for those that will change, but also give
    313             # the AttributeError

AttributeError: module 'numpy' has no attribute '_no_nep50_warning'

[NivekT]

Thanks for flagging that!

At a quick glance, I can't tell if it an issue with our library. Do you have more information? Are you running on Colab such that you can provide a reproducible script?

It may be a numpy issue similar to this, can you update your numpy package? You may need a version > 1.21 .

As a workaround, you may be able to import the specific utils function that you need, rather than the whole submodule.

[NivekT]

Hi @ayrtondenner, since this feature has been added. I will close this for now. Don't hesitate to comment or reach out if anything is not working for you!