search

Kong / gateway-operator

Kubernetes Operator for Kong Gateways
Apache License 2.0
49 stars 11 forks source link

AIGateway Support #137

Open shaneutt opened 7 months ago

shaneutt commented 7 months ago

Problem Statement

We currently have several plugins for the Kong Gateway which enable access to large language models (LLMs):

https://github.com/Kong/kong/pull/12341 https://github.com/Kong/kong/pull/12340 https://github.com/Kong/kong/pull/12337 https://github.com/Kong/kong/pull/12336

When applied to a Gateway they create our "AI Gateway" for serving and managing traffic to well-known LLM providers like OpenAI's ChatGPT.

If you're using the KGO and your goal is to try out the "AI Gateway", there's a lot of configuration that you need to do manually, and CRDs you have to learn like KongPlugin.

Proposed Solution

Create an AIGateway CRD and controller which automates this configuration and management for the user with minimal upfront configuration. Ideally the user should just be able to add their API keys to a Secret, and then the rest of the YAML can just be applied in a one-liner, resulting in an endpoint where they can access their AI Gateway in the AIGateway.status.

This should start out as an experimental and gated feature, so that we can share it with users and incorporate feedback into its development.

Additional Information

This issue originates from Aha! KIC-I-53.

Acceptance Criteria

### Tasks
- [ ] https://github.com/Kong/gateway-operator/pull/1396
- [ ] https://github.com/Kong/gateway-operator/pull/1399
- [ ] https://github.com/Kong/gateway-operator/issues/1429
- [ ] https://github.com/Kong/gateway-operator/issues/1430
- [ ] https://github.com/Kong/gateway-operator/pull/1573
- [ ] Use GenerateName for owned resources
- [ ] Implement update/patch management for sub-resources
- [ ] Expand watch capabilities to map AIGateways to owned resource changes
- [ ] Implement full status handling
- [ ] Enable real integration tests?
shaneutt commented 7 months ago

Met with @mflendrich and @mheap this morning. We are going to make this an OSS feature, and we're going to try for release in v1.2.0 (if that doesn't pan out for some reason, v1.3.0). The team doesn't have capacity by itself to do this during these release cycles based on current commitments, so I'm going to join the team to help with this one.

Note: This feature requires Gateway v3.6.0, which we currently expect to release February 12th, however this shouldn't block development as the relevant features are available in current release candidates.

/cc @tysoekong

mheap commented 7 months ago

We don't have a confirmed date for v1.3.0, so I'd recommend pushing for v1.2.0