Investiate: cleanup listeners

katanemo / archgw

Arch is the intelligent gateway for AI agents. Engineered with (fast) LLMs for the secure handling, robust observability, and seamless integration of prompts with your APIs - outside business logic. Built by the core contributors of Envoy proxy, on Envoy.

https://archgw.com

Apache License 2.0

796 stars 35 forks source link

Investiate: cleanup listeners #299

Open adilhafeez opened 1 day ago

adilhafeez commented 1 day ago

We should have two listeners one for prompt and other for llm routing.

Here is a list of listeners we have today,

10000 prompt (to create traceparent)
10001 prompt - hosts prompt gatreway wasm filter code
11000 internal api - to communicate with internal hosts like embeddings, nli, archfc etc.
12000 llm routing (to create traceparent)
12001 llm routing - hosts llm gateway filter code

adilhafeez commented 1 day ago

review async client and come up with recommendation