ketangangal
commented
1 year ago Hello @kf-rahman ,
Its very handy to use for LLMs all you need to do is Generate embeddings with you llm model and use evidently for monitoring. You can also refer : https://docs.evidentlyai.com/user-guide/customization/embeddings-drift-parameters
FYI @elenasamuylova
Hi @kf-rahman
If you run LLM models in production, you can use Evidently to evaluate and monitor input text data quality, text data drift, and embedding drift.
We do not have any LLM-specific evaluation metrics yet, but depending on the task, you can also use Evidently to evaluate the model quality (e.g., use Classification Presets if you work on text classification tasks).
Which use case do you have in mind?