confident-ai / deepeval

The LLM Evaluation Framework
https://docs.confident-ai.com/
Apache License 2.0
3.68k stars 292 forks source link

Enhancing DeepEval's Vertex AI implementation sample #881

Open meteatamel opened 4 months ago

meteatamel commented 4 months ago

DeepEval docs have a Google Vertex AI implementation sample here: https://docs.confident-ai.com/docs/metrics-introduction#google-vertexai-example

However, it has a number of issues:

  1. It doesn't seem to subclass the base class correctly
  2. It unnecessarily relies on LangChain to call Vertex AI
  3. It unnecessarily exposes the underlying library and its safety settings outside the GoogleVertexAI class

I implemented another version here that fixes these issues: https://github.com/meteatamel/genai-beyond-basics/tree/main/samples/evaluation/deepeval/vertex_ai

I have the two variants: one that uses Vertex AI directly, one that uses Vertex AI via LangChain.

I also have 3 samples how to use it here: https://github.com/meteatamel/genai-beyond-basics/tree/main/samples/evaluation/deepeval

I'm hoping the DeepEval documentation can be updated to use these implementations instead.

penguine-ip commented 4 months ago

@meteatamel Thanks for the comments, will definitely update it :)