search

alkem-io / virtual-contributor-engine-guidance

Guidance on usage of the Alkemio platform based on GenAI
European Union Public License 1.2
3 stars 0 forks source link

LLM API calls cost management #48

Open renehonig opened 10 months ago

renehonig commented 10 months ago

Is your feature request related to a problem? Please describe. As an operations person I want to be sure that the use of LLM calls to external API's does not become cost prohibitive.

Describe the solution you'd like The guidance engine uses an external service provider for LLM inference for which costs are charged to Alkemio. Whilst individual calls are relatively cheap, the costs could still become prohibitive if the usage patterns are different from what is expected, e.g. due to a bug or some type of misuse by the user. The majority of controls should be managed through the server, but it is probably prudent to build in some simple reporting/rate limiting on the guidance engine side.

Describe alternatives you've considered Full control of the API calls to the engine on the server side.

Additional context Over time, more granular controls may shift to the server side. Also, the use of LLM is expected to grow, which will create additional/different requiremenrts for cost management.

techsmyth commented 9 months ago

@renehonig moving this + other guidance engine issues for new functionality to icebox pending the wider Roadmap prioritization.