flowaicom / flow-judge

Code for evaluating with Flow-Judge-v0.1 - an open-source, lightweight (3.8B) language model optimized for LLM system evaluations. Crafted for accuracy, speed, and customization.
Apache License 2.0
53 stars 8 forks source link

refactored langchain integration and tutorial #15

Closed vaahtio closed 1 month ago

vaahtio commented 1 month ago

LangChain Integration for Flow Judge

Summary

This PR introduces an integration between Flow Judge and LangChain, allowing users to leverage Flow Judge's custom metrics within LangChain workflows.

Key Changes

  1. Created FlowJudgeLangChainEvaluator class on the integrations folder:

    • Extends LangChain's StringEvaluator
    • Enables use of Flow Judge metrics in LangChain pipelines
    • Refactored to the changes in main branch mainly related to model classes
  2. Added example notebook:

    • Demonstrates usage of Flow Judge integration within LangChain
    • Compares Flow Judge custom metrics with LangChain's built-in QA evaluator

Testing

The integration has been manually tested:

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 0% with 51 lines in your changes missing coverage. Please review.

:white_check_mark: All tests successful. No failed tests found.

Files with missing lines Patch % Lines
flow_judge/integrations/langchain.py 0.00% 51 Missing :warning:
Files with missing lines Coverage Δ
flow_judge/integrations/langchain.py 0.00% <0.00%> (ø)

... and 9 files with indirect coverage changes