Configure OpenAI endpoint

[douglasg14b]

There are many tools/services that provide "OpenAI Compatable" endpoints in a bring-your-own-ai sort of way. Even moreso in corporate environments where we may have GPT-4 completion access that don't share or store corporate data provided on different endpoints (ie. From Azure).

Can the OpenAI endpoint be configurable so it transparently "just works" with any OpenAI-compatable endpoint?

[C-Loftus]

Yes that is definitely feasible and probably relatively easy. if you want to submit a PR, you could just create a setting to change the following from a string to a talon setting https://github.com/C-Loftus/talon-ai-tools/blob/62a1ad117ef5433462eb6dddb59bd6150683688f/GPT/gpt.py#L64 and then the user can specify the Talon setting for the url in a talon file

only thing I will say is sometimes the default system prompt is a bit different, for instance for llamafiles I have to specify ( I don't know if this is a similar thing with azure I do not use that at all)

'messages': [
                    {
                        "role": "system",
                        "content": "You are an assistant helping an office worker to be more productive."
                    },
                    {
                        'role': 'user', 
                        'content': f"{prompt}:\n{content}"
      }

[douglasg14b]

Thanks for the reply!!

For Azure at least they provide direct access to OpenAI models for enterprises. So it's the same model.

Is there any weirdness if this points at an endpoint of that uses GPT-4?

[C-Loftus]

I don't think so, but I'm honestly not sure since I haven't tested much with GPT-4 in general. (For personal use, a lot of people don't get access to it unless they have paid over a certain threshold, which my usage is not at yet.)

If you create a pull request to support this, I would be happy to merge. That being said, I will probably not implement this on my own since I don't have any endpoints to test with.

This is generally a good idea though and would help reduce some of the duplicate code I have for the llamafile anyways.

[C-Loftus]

@douglasg14b I believe I generalized this to support Azure and any compatible endpoint in my latest PR. Can you check out https://github.com/C-Loftus/talon-ai-tools/pull/24 and see if it works with your azure endpoint? (you can use the gh cli tool to do this very easy if you aren't familiar, then all you need to do is set the user.model_endpoint setting somewhere in a talon file) If you can't test that is okay, since I tested and it works with local llamafiles, so at the very least that pull request is not a regression

[C-Loftus]

Closing since I think this is addressed. Will reopen if we get the chance to test and find it doesn't work for some reason