[Sub-issue] [MultiModel support] API Key Update Endpoint

[mmabrouk]

Context:

We would like to enable users to update their LLM API keys in agenta and continue using their apps with the new API keys. Right now, after an app is created it is impossible to update the API key (other than by reserving the application from code with a new .env)

Suggested solution:

• We would add a new argument to /generate and /generate_deployed endpoints that takes environment variable. When these are passed, our SDK would overwrite the environment variables that are set (either in the .env or when starting the container).
• When we call the app from the playground, we would always inject any API key that is provided in the UI to the calls to /generate
• In the deployment view, we would update the snippet to show that the user needs to add the API keys of the LLMs that they are using

Other solutions considered:

• Saving the LLM keys in the backend: Two issues there 1) security and 2) adding latency to calls (before each call we would need to grab the LLM key, although this can be resolved in caching and retrial). The first issue is the main issue
• Adding an endpoint to update the API keys in the env variable: overly complex for no advantage

The disadvantage of this solution is: 1) The requests for generate_deployed would require an api key in the call and therefore cannot be called directly from a frontend=> Not a real disadvantage, since anyhow, in a real production system, our LLM apps would also require API keys in the call, and would also not be called from the frontend. 2) Sharing LLM keys between users becomes difficult: Many organization would like to share API keys without giving the whole team access to the API keys. To implement this, we would need to 1) save the keys in our backend 2) implement the logic to sharing it between the configuration anyhow

Overall, I think the solution suggested is 1) simple to implement 2) solves the problem quite well for now. Users can even update the keys for their code served apps.

@aakrem @aybruhm What do you think? Did I miss something?

[mmabrouk]

@mmabrouk evaluation

[aakrem]

• I think reserving the apps is still the right solution.
• I think supporting reading env vars from both os env and fetching them from an endpoint, and to dynamically update the env vars changes from the endpoint, introduces a layer of complexity that we don't need for now. However I suggest enhancing both cli and ui to give users better experience to restart many apps.