search

sigoden / aichat

All-in-one LLM CLI tool featuring Shell Assistant, Chat-REPL, RAG, AI tools & agents, with access to OpenAI, Claude, Gemini, Ollama, Groq, and more.
Apache License 2.0
4.44k stars 300 forks source link

Performance stats option #481

Closed kir-gadjello closed 6 months ago

kir-gadjello commented 6 months ago

Please add basic performance stats: prompt processing tokens/s, generation tokens/s behind a key like -vs Also a mode for debugging LLM API requests (log as json) would be useful behind a key like -va

sigoden commented 6 months ago

Why a personal command-line tool needs this?

kir-gadjello commented 6 months ago

It is useful when you use it with local models to find the best configuration

sigoden commented 6 months ago

These stats should not be botained from aichat, but should be obtained from the tools that directly run the model (such as ollama, localai). For example, for many models, what they return through streaming api is not a token, but a large segment. Its statistics are distorted.