CRITICAL BUG: images.generate does not work on Azure OpenAI

[nicolaschaillan]

        openai_api_version = "2023-09-01-preview"
        client = AzureOpenAI(azure_endpoint = openai_api_base, api_key=openai_api_key, api_version=openai_api_version)

        response = client.images.generate(
            prompt=prompt,
            size=size,
            n=n
        )

Fails with:

{'error': {'code': '404', 'message': 'Resource not found'}}

[kristapratico]

Hi @nicolaschaillan, Azure OpenAI Dall-E is not supported in the v1 SDK yet, but support will be added soon.

https://learn.microsoft.com/en-us/azure/ai-services/openai/dall-e-quickstart?tabs=command-line&pivots=programming-language-python#install-the-python-sdk

[nicolaschaillan]

This should be documented. Once folks migrate it's impossible to go back with all those changes and now I'm left with no option for our DALL E deployment... Not good! Any ETA for the support?

We have hundreds of users on Azure OpenAI using it.

[kristapratico]

@nicolaschaillan I will share as soon as I have an ETA. As a stop-gap, it is possible to access Azure OpenAI DALL-E by passing in a httpx custom transport:

import time
import json
import httpx
import openai

class CustomHTTPTransport(httpx.HTTPTransport):
    def handle_request(
        self,
        request: httpx.Request,
    ) -> httpx.Response:
        if "images/generations" in request.url.path and request.url.params[
            "api-version"
        ] in [
            "2023-06-01-preview",
            "2023-07-01-preview",
            "2023-08-01-preview",
            "2023-09-01-preview",
            "2023-10-01-preview",
        ]:
            request.url = request.url.copy_with(path="/openai/images/generations:submit")
            response = super().handle_request(request)
            operation_location_url = response.headers["operation-location"]
            request.url = httpx.URL(operation_location_url)
            request.method = "GET"
            response = super().handle_request(request)
            response.read()

            timeout_secs: int = 120
            start_time = time.time()
            while response.json()["status"] not in ["succeeded", "failed"]:
                if time.time() - start_time > timeout_secs:
                    timeout = {"error": {"code": "Timeout", "message": "Operation polling timed out."}}
                    return httpx.Response(
                        status_code=400,
                        headers=response.headers,
                        content=json.dumps(timeout).encode("utf-8"),
                        request=request,
                    )

                time.sleep(int(response.headers.get("retry-after", 10)))
                response = super().handle_request(request)
                response.read()

            if response.json()["status"] == "failed":
                error_data = response.json()
                return httpx.Response(
                    status_code=400,
                    headers=response.headers,
                    content=json.dumps(error_data).encode("utf-8"),
                    request=request,
                )

            result = response.json()["result"]
            return httpx.Response(
                status_code=200,
                headers=response.headers,
                content=json.dumps(result).encode("utf-8"),
                request=request,
            )
        return super().handle_request(request)

client = openai.AzureOpenAI(
    azure_endpoint="<azure_endpoint>",
    api_key="<api_key>",
    api_version="<api_version>",
    http_client=httpx.Client(
        transport=CustomHTTPTransport(),
    ),
)
image = client.images.generate(prompt="a cute baby seal")

print(image.data[0].url)

Once support is officially added, you can remove the http_client keyword argument and the custom transport class from your code and all should work as expected. Let me know if that helps.

[nicolaschaillan]

Thanks!

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Krista Pratico @.> Sent: Monday, November 6, 2023 9:16:52 PM To: openai/openai-python @.> Cc: Nicolas Chaillan @.>; Mention @.> Subject: Re: [openai/openai-python] CRITICAL BUG: images.generate does not work on Azure OpenAI (Issue #692)

@nicolaschaillanhttps://github.com/nicolaschaillan I will share as soon as I have an ETA. As a stop-gap, it is possible to access Azure OpenAI DALL-E by passing in a httpx custom transport:

import time import json import httpx import openai

class CustomHTTPTransport(httpx.HTTPTransport): def handle_request( self, request: httpx.Request, ) -> httpx.Response: if "images/generations" in request.url.path and request.url.params[ "api-version" ] in [ "2023-06-01-preview", "2023-07-01-preview", "2023-08-01-preview", "2023-09-01-preview", "2023-10-01-preview", ]: request.url = request.url.copy_with(path="/openai/images/generations:submit") response = super().handle_request(request) operation_location_url = response.headers["operation-location"] request.url = httpx.URL(operation_location_url) request.method = "GET" response = super().handle_request(request) response.read()

        timeout_secs: int = 120
        start_time = time.time()
        while response.json()["status"] not in ["succeeded", "failed"]:
            if time.time() - start_time > timeout_secs:
                raise openai.OpenAIError("Operation polling timed out.")

            time.sleep(int(response.headers.get("retry-after")) or 10)
            response = super().handle_request(request)
            response.read()

        if response.json()["status"] == "failed":
            error_data = response.json()["error"]
            message = error_data.get("message", "Operation failed")
            code = error_data.get("code")
            raise openai.OpenAIError(message, code)

        result = response.json()["result"]
        return httpx.Response(
            status_code=200,
            headers=response.headers,
            content=json.dumps(result).encode("utf-8"),
            request=request,
        )

client = openai.AzureOpenAI( azure_endpoint="", api_key="", api_version="", http_client=httpx.Client( transport=CustomHTTPTransport(), ), ) image = client.images.generate(prompt="a cute baby seal") print(image.data[0].url)

Once support is officially added, you can remove the http_client keyword argument and the custom transport class from your code and all should work as expected. Let me know if that helps.

— Reply to this email directly, view it on GitHubhttps://github.com/openai/openai-python/issues/692#issuecomment-1797181743, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABK2VBVJHUV4EEZ35TIUMQ3YDGKZJAVCNFSM6AAAAAA7AJI3A6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJXGE4DCNZUGM. You are receiving this because you were mentioned.Message ID: @.***>

[ashjeanbird]

Hey @kristapratico, I made an update to the docs per @nicolaschaillan suggestion. Can you please take a look to see if it works and if there is anything else that I should add? Happy to add any additional details.

[StephenHodgson]

It is because msft adds an additional :submit part to the end of the uri they specify for generation. You can easily append it to your endpoint request as needed.

I also added feedback to msft to remove this uri part:

https://feedback.azure.com/d365community/idea/f2204961-d97f-ee11-a81c-6045bdb7ea56

[kristapratico]

@ashjeanbird I'll take a look.

@StephenHodgson yes, the Azure Dall-E endpoint contains a :submit for the initial call. However, it's not enough to just append this to the endpoint. The initial call returns a status monitor which has to be checked until the final result is ready. This code shows how to workaround this for now: https://github.com/openai/openai-python/issues/692#issuecomment-1797181743

[StephenHodgson]

Yeah I don't understand why Microsoft went so far different with this endpoint implementation. Super frustrating.

[ashjeanbird]

@kristapratico thank you for checking out my PR https://github.com/openai/openai-python/pull/766. I have made a change to reflect the known issues verbiage link and the associated workaround.

[kristapratico]

@nicolaschaillan @StephenHodgson @ashjeanbird Azure OpenAI now supports Dall-E 3. See this Quickstart for details. It is supported on version 1.2.4 or greater of the openai library.

Please note that if you wish to continue to use Dall-E 2 you must either use the 0.28.1 version of the library or use 1.X version with the workaround.

[nicolaschaillan]

@nicolaschaillan @StephenHodgson @ashjeanbird Azure OpenAI now supports Dall-E 3. See this Quickstart for details. It is supported on version 1.2.4 or greater of the openai library.

Please note that if you wish to continue to use Dall-E 2 you must either use the 0.28.1 version of the library or use 1.X version with the workaround.

This doesn't work whatsoever on my East US API...

{'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

Any clue why?

[kristapratico]

Are you using Dall-E 2 or 3?

Dall-E 3 is currently only available on SwedenCentral: https://learn.microsoft.com/en-us/azure/ai-services/openai/dall-e-quickstart?tabs=dalle3%2Ccommand-line&pivots=programming-language-python

You'll need to deploy the dall-e-3 model (you can do this from the Azure OpenAI Studio) and then target the deployment name in the method call:

result = client.images.generate(
    model="dalle3", # the name of your DALL-E 3 deployment
    prompt="a close-up of a bear walking through the forest",
)

[nicolaschaillan]

I tried both. Both don't work.

I use openai_api_version = "2023-12-01-preview"

model = "dalle2"

Fails.

We can't use Sweden for the U.S government work obviously, this should be better documented... list of supported models should be in a single location and easy to find. This is just a plain mess/nightmare.

[nicolaschaillan]

Btw on East, I don't see a way to add DALL-E but yet the example code is there and I can generate images in the API with the old code (without the new client)

[kristapratico]

I'm sorry for the confusion. I'm relaying your feedback to the team.

For clarity,

Azure OpenAI supports two models for image generation: Dall-E 2 and Dall-E 3

Dall-E 2 is available in East US. You can use version 0.28.1 of the openai library to access it. If you want to use this model version with v1.X of the openai library, then you can copy the code from the workaround.

Dall-E 3 is currently only available in SwedenCentral. You can use v1.2.4+ of the openai library to access it. Note that you must first deploy the dall-e-3 model from the Azure OpenAI studio and pass your deployment name in the model parameter when calling the API.

I am reaching out, but not sure yet when Dall-E 3 will be supported in US regions. The model documentation will be updated when it becomes available in more regions: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#dall-e-models-preview

[nicolaschaillan]

Thanks, all these breaking changes because of MSFT are quite annoying, not going to lie. Why can't I use the new 1.2.4+ on DALL-E in East region?! Seems like it should be quite simple?

[StephenHodgson]

I'm guessing whenever this goes in https://github.com/Azure/azure-rest-api-specs/pull/26582

[brunobraga]

I am having this issue as well... If helps anyone, solution below.

Code that I expected to work (Dall-E 2 - US East):

client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    azure_endpoint=f'https://{os.getenv("AZURE_OPENAI_API_INSTANCE_NAME")}.openai.azure.com',
    api_version=os.getenv('AZURE_OPENAI_API_VERSION'),
)
response = client.images.generate(
    prompt='dancing cat',
    size='512x512',
    n=1
)
image_url = response.data[0].url

Alternative code that works fine (Dall-E 2):

url = "{}/openai/images/generations:submit?api-version={}".format(
    f'https://{os.getenv("AZURE_OPENAI_API_INSTANCE_NAME")}.openai.azure.com',
    os.getenv('AZURE_OPENAI_API_VERSION')
)
headers= { "api-key": os.getenv("AZURE_OPENAI_API_KEY"), "Content-Type": "application/json" }
body = {
    "prompt": "dancing cat",
    "n": 1,
    "size": '512x512'
}
submission = requests.post(url, headers=headers, json=body)
operation_location = submission.headers['Operation-Location']
status = ""
while (status != "succeeded"):
    time.sleep(3) # workaround rate limit
    response = requests.get(operation_location, headers=headers)
    status = response.json()['status']
image_url = response.json()['result']['data'][0]['url']

Code currently working for Dall-E 3 (Sweden Central):

client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    azure_endpoint=f'https://{os.getenv("AZURE_OPENAI_API_INSTANCE_NAME")}.openai.azure.com',
    api_version=os.getenv('AZURE_OPENAI_API_VERSION'),
)
response = client.images.generate(
    prompt='dancing cat',
    model='dall-e-3',
    n=1
)
image_url = response.data[0].url

[cohml]

@kristapratico Is this workaround (i.e., implementing CustomHTTPTransport) still required for accessing DALL-E via Azure OpenAI?

[kristapratico]

@kristapratico Is this workaround (i.e., implementing CustomHTTPTransport) still required for accessing DALL-E via Azure OpenAI?

The workaround is only necessary if you're using Azure with DALL-E 2. Azure DALL-E 3 is supported by the library.

[cohml]

The workaround is only necessary if you're using Azure with DALL-E 2. Azure DALL-E 3 is supported by the library.

Thanks for the quick reply.

Do you know if that is set to change at some point? Or if you're forever stuck with DALL-E 2, are you forever stuck with CustomHTTPTransport?

[kristapratico]

The workaround is only necessary if you're using Azure with DALL-E 2. Azure DALL-E 3 is supported by the library.

Thanks for the quick reply.

Do you know if that is set to change at some point? Or if you're forever stuck with DALL-E 2, are you forever stuck with CustomHTTPTransport?

As far as I know, there won't be updates to Azure DALL-E 2 in the future, meaning the workaround will be necessary to use it with this library. Are you able to migrate to DALL-E 3?

[StephenHodgson]

Do you know if that is set to change at some point? Or if you're forever stuck with DALL-E 2, are you forever stuck with CustomHTTPTransport?

As far as I know, there won't be updates to Azure DALL-E 2 in the future, meaning the workaround will be necessary to use it with this library. Are you able to migrate to DALL-E 3?

Good to know so I can document this in my client libraries.

[ronaldchoi]

@nicolaschaillan I will share as soon as I have an ETA. As a stop-gap, it is possible to access Azure OpenAI DALL-E by passing in a httpx custom transport:

import time
import json
import httpx
import openai

class CustomHTTPTransport(httpx.HTTPTransport):
    def handle_request(
        self,
        request: httpx.Request,
    ) -> httpx.Response:
        if "images/generations" in request.url.path and request.url.params[
           ...

Once support is officially added, you can remove the http_client keyword argument and the custom transport class from your code and all should work as expected. Let me know if that helps.

This workaround does not work with my Azure setup and always hits the "[429 Too Many Requests]" errors.

I was able to fix it by adding a default value for the "retry-after" header in case it doesn't exist:

time.sleep(int(response.headers.get("retry-after", 10)))

[kristapratico]

@ronaldchoi thanks for spotting the bug. I'll update the example in my comment.