confident-ai / deepeval

The LLM Evaluation Framework
https://docs.confident-ai.com/
Apache License 2.0
2.9k stars 213 forks source link

Enhancing DeepEval's Vertex AI implementation sample #881

Open meteatamel opened 1 month ago

meteatamel commented 1 month ago

DeepEval docs have a Google Vertex AI implementation sample here: https://docs.confident-ai.com/docs/metrics-introduction#google-vertexai-example

However, it has a number of issues:

  1. It doesn't seem to subclass the base class correctly
  2. It unnecessarily relies on LangChain to call Vertex AI
  3. It unnecessarily exposes the underlying library and its safety settings outside the GoogleVertexAI class

I implemented another version here that fixes these issues: https://github.com/meteatamel/genai-beyond-basics/tree/main/samples/evaluation/deepeval/vertex_ai

I have the two variants: one that uses Vertex AI directly, one that uses Vertex AI via LangChain.

I also have 3 samples how to use it here: https://github.com/meteatamel/genai-beyond-basics/tree/main/samples/evaluation/deepeval

I'm hoping the DeepEval documentation can be updated to use these implementations instead.

penguine-ip commented 1 month ago

@meteatamel Thanks for the comments, will definitely update it :)