kubectl plugin for AIGateway

[shaneutt]

Problem Statement

The AIGateway effort (see Kong/gateway-operator#137) adds a bespoke Gateway implementation with automatic configuration and lifecycle management to connect to (and manage inference for) AI models such as Large Language Models (LLM).

It doesn't take much work to configure an AIGateway (as automating and making that simple is the entire purpose of the feature), mostly just YAML configuration which works great for gitops pipelines, e.t.c. For demos or quick use however, it could be helpful to have a command line interface.

Proposed Solution

Create a Kubectl plugin to create AIGateways:

$ kubectl ai-gateway create
Please enter a provider (e.g. OpenAI, Cohere, Azure): OpenAI
Please enter the API token for the provider: **************************************
Do you need to add any more providers <Y/N>? n
SUCCESS! AIGateway named "kong-ai-gateway" has been created in namespace "default"

And then expose models:

$ kubectl ai-gateway model add devteam-chatgpt \
  --provider openai \
  --model gpt-3.5-turbo \
  --default-prompts '{"role":"system","content":"you are a helpful assistant who responds in the style of Sherlock Holmes"}'
SUCCESS! "devteam-chatgpt" has been created.

Perform inference (where we will automatically retrieve the consumer and handle auth, e.t.c.):

$ kubectl ai-gateway model chat "Provide an example of making an HTTP request in Rust" --identifier devteam-chatgpt
Indubitably, my good sir:

---
use reqwest::Error;

#[tokio::main]
async fn main() -> Result<(), Error> {
    let url = "http://httpbin.org/get";
    let res = reqwest::get(url).await?;
    if res.status().is_success() {
        let body = res.text().await?;
        println!("Response Text: {}", body);
    } else {
        println!("Request failed with status: {}", res.status());
    }
    Ok(())
}
---

And manage adding new providers and models later:

$ kubectl ai-gateway provider add azure
Please enter the Azure API token: ********************************
SUCCESS! Provider added
$ kubectl ai-gateway models add devteam-phi-2 --provider azure
SUCCESS! "devteam-phi-2" has been created.

Additional Information

This seems very strongly in the nice-to-have territory, we should probably wait to see how popular the AIGateway feature is before prioritizing this.

Blocked until Kong/gateway-operator#137 is complete.

Consider documenting installation with krew to simplify documentation.

Acceptance Criteria

• [ ] as a user I can create and manage my AIGateways and perform inference using a command line interface

[shaneutt]

/cc @tysoekong