[Feature Request] Private Streaming and Backend APIs

[ggallo]

Describe the solution you'd like

Create a deployment option that bundles all backend

Why the solution needed

Many customers require any and all internal usage of generative AI to transit their corporate network. At present the use of HTTP API and Websocket API means that using this solution as-is is a non-starter.

Options include:

• Converting HTTP API to a REST API, thus allowing use of private API (done)
• Convert the Streaming API to an ECS Service using websockets

Additional context

You can reach me on Slack

Implementation feasibility

Are you willing to discuss the solution with us, decide on the approach, and assist with the implementation?

• [X] Yes!
• [ ] No

[imvijith]

@statefb is there a plan to support websocket based streaming for api publish stack