Just Questions of guidance and Arch and configuration settings

[ehaken95]

Hello, I am a developer learning AI Gateway. I have an interest in the arch solution, and as I explore the source code and demo version, I have a few questions.

1. While reviewing the documentation, I noticed the mention of "Arch binds itself to a local address such as 127.0.0.1:9000/v1 or a DNS-based address like arch.local:9000/v1 for outgoing traffic" regarding upstream. Where and how should I configure this setting to allow such configuration?

2. I followed the pre-requisites and the instructions in function_calling > README.md for the demo version, but I encountered an API connection error in AI chat (I did set the OPENAI_API_KEY). To address this, I modified the address in function_calling > docker-compose.yaml as shown below, which made the chat work but only resulted in simple OpenAI chat, rather than the example-based chat functionality. Could you provide additional guidance on this?

chatbot_ui:
  ...
   environment:
     - CHAT_COMPLETION_ENDPOINT=https://api.openai.com/v1
     #- CHAT_COMPLETION_ENDPOINT=http://host.docker.internal:12000/v1
  ...

[salmanap]

Thanks a ton for reporting these issues. Comments in-line

Hello, I am a developer learning AI Gateway. I have an interest in the arch solution, and as I explore the source code and demo version, I have a few questions.

1. While reviewing the documentation, I noticed the mention of "Arch binds itself to a local address such as 127.0.0.1:9000/v1 or a DNS-based address like arch.local:9000/v1 for outgoing traffic" regarding upstream. Where and how should I configure this setting to allow such configuration?

I believe you are talking about this section. https://docs.archgw.com/concepts/tech_overview/listener.html? The ports mentioned there are incorrect. That's a bug. Arch listens by default on 10000 for incoming prompts and 12000 for outgoing calls to LLMs. The 9000 port is used for admin stats. I'll fix that asap: https://github.com/katanemo/arch/issues/252

2. I followed the pre-requisites and the instructions in function_calling > README.md for the demo version, but I encountered an API connection error in AI chat (I did set the OPENAI_API_KEY). To address this, I modified the address in function_calling > docker-compose.yaml as shown below, which made the chat work but only resulted in simple OpenAI chat, rather than the example-based chat functionality. Could you provide additional guidance on this?

chatbot_ui:
  ...
   environment:
     - CHAT_COMPLETION_ENDPOINT=https://api.openai.com/v1
     #- CHAT_COMPLETION_ENDPOINT=http://host.docker.internal:12000/v1
  ...

Please update the config to CHAT_COMPLETION_ENDPOINT=http://host.docker.internal:10000/v1. The 10000/v1 is for handling incoming requests.

Can you please try this again and let me know if you can get the demo up?

[ehaken95]

Thank you for your patience in awaiting my response. Here are my answers regarding the two points you mentioned:

1. Thank you for confirming. Currently, I’m reviewing whether the upstream call to the arch gateway involves communication with an external DNS.

To clarify, the flow follows this sequence:

user > 
   arch > 
      other system URL (e.g., http://example.com/mngt/v1/arch/message) > 
         DO SOMETHING > 
           arch > 
              user

In the demo setup, the API call was made directly in the Python code. However, we aim to manage this using YAML instead of writing or modifying the Python code. This requires verifying whether the AI can call URLs from other locations.

2. Thanks for the suggestion. Switching to port 10000 enabled the demo to function properly.