search

edubruell / tidyllm

An tidy interface to large language model APIs for R
https://edubruell.github.io/tidyllm/
Other
36 stars 2 forks source link

Think about a way to handle model call metadata in the LLMMessage after an API function was run #21

Closed edubruell closed 2 weeks ago

edubruell commented 1 month ago

Description

Enhance the LLMMessage class to capture and store model call metadata (such as model parameters and tokens used during the API call) after running a function like chatgpt(). This addition will enable users to inspect details about the underlying model interaction and provide insight into the model’s performance and usage, including tracking token consumption, response times, or other relevant metadata from the API reply.

Possible Actions

  1. Extend LLMMessage class to store metadata:

    • Add a new field, model_call_metadata, to store metadata for each message. This field will store details such as the model used, token count, parameters applied, and API-specific metadata (e.g., latency or cost estimation).
    • Modify the add_message() function to accept an optional metadata parameter, allowing it to attach model call information when appending a new message.

  2. Automatically capture metadata after API calls:

    • Modify API interaction methods (e.g., chatgpt(), claude(), groq(), mistral() and ollama()) to capture relevant model metadata after the call, including type of API, type of model, tokens used, temperature, model version, etc.
    • After each API call, update the LLMMessage object to store the metadata along with the generated message.

  3. Add get_metadata() function:

    • Implement a function to retrieve metadata from the message history for inspection. This function should support filtering metadata by message type (e.g., only assistant responses made by claude()) and providing a summary of total tokens used across the conversation.
    • Provide options for formatting metadata output for easier integration into reporting or logging systems.

  4. Update printing methods:

    • Update the print() method to optionally display metadata alongside each message, including tokens used or model-specific details, to give users deeper insight when reviewing a conversation history.

  5. Fallback mechanism for incomplete metadata:

    • Implement a mechanism to handle cases where metadata may be unavailable (e.g., unsupported APIs or failed calls) by setting default values or warnings indicating incomplete data.
edubruell commented 2 weeks ago

new metadata function that works both with sync chat requests and batch requests.