Set up a proxy for OpenAI.

[shouyezhe]

Recently, I tried to use OpenAI's API in LMQL, but I couldn't find an option to set up a proxy in LMQL. I know that in OpenAI, you can use 'openai.proxy' to set it up, but how can I do it in LMQL?

[lbeurerkellner]

There is an endpoint parameter, e.g.

lmql.model("openai/<model>", endpoint="<endpoint>")

This is meant to be used with mock implementations of the OpenAI API. Are you looking for this way of configuring it, or are you looking for a traditional network-level way of proxying requests?

Let me know what concrete Proxy/mock implementation you want to use, so I can have a look at general compatibility.

[shouyezhe]

I'm using Python to call LMQL with the @lmql.query decorator. I attempted to provide the endpoint parameter in the lmql.query() function and execute it, but OpenAI returned a TimeoutError. When I used the same port settings and called OpenAI directly through openai.proxy, everything worked fine. This led to my previous question.

[lbeurerkellner]

I understand, thanks for clarifying. I looked up the proxy parameter and found this description in the official OpenAI client library https://github.com/openai/openai-python#command-line-interface.

Since we currently do not use this library, we unfortunately cannot directly pass the parameter through. So for now, I have to say that proxy is not supported. However, we want to adopt the official API library soon, which should enable this. I will keep this issue around to track progress on this.

[shouyezhe]

Thanks for your work. This enhancement is crucial for my smooth use of LMQL. I hope to see it soon. Best wishes.

[lbeurerkellner]

Marking this as a good first issue.

Similar to decoding parameters like top_k (e.g. as tracked here https://github.com/eth-sri/lmql/blob/main/src/lmql/runtime/openai_integration.py#L135), proxy could also just be passed through from the lmql.model(...) call, or the decoder arguments.

[Saibo-creator]

I'm not completely sure I've got a handle on the issue. It looks like the proxy is functioning correctly with the endpoint parameter. Maybe the issue was related to that the proxy mode does not set the secret.

import lmql

# uses 'chatgpt' by default
@lmql.query(model="chatgpt")
def tell_a_joke():
    '''lmql
    """A great good dad joke. A indicates the punchline
    Q:[JOKE]
    A:[PUNCHLINE]""" where STOPS_AT(JOKE, "?") and \
                           STOPS_AT(PUNCHLINE, "\n")
    '''

tell_a_joke() # uses chatgpt
 tell_a_joke(model=lmql.model("openai/gpt-4", endpoint="https://api.openai-proxy.com/v1/chat/completions")) #using a proxy  https://api.openai-proxy.com which provide the same API as openai 

LMQLResult(prompt="A great good dad joke. A indicates the punchline\nQ: Why don't scientists trust atoms?\nA: Because they make up everything!", variables={'JOKE': " Why don't scientists trust atoms?", 'PUNCHLINE': ' Because they make up everything!'}, distribution_variable=None, distribution_values=None, sequence=<seq token_len=31 ids=[... b'\n', b'A', b':', b' Because', b' they', b' make', b' up', b' everything', b'!', b'<|endoftext|>']>)

If the request is to expose the proxy argument in the decoding function, yes this is definitely doable. But directly initializing the model with the proxy seems more user-friendly since it avoids having to specify the proxy URL in every call.

What do you think ? :thinking: