langchain-ai / langsmith-sdk

LangSmith Client SDK Implementations
https://smith.langchain.com/
MIT License
387 stars 72 forks source link

Issue: Evaluator raises Invalid run type error #582

Closed DaveOkpare closed 3 weeks ago

DaveOkpare commented 5 months ago

Issue you'd like to raise.

Evaluator on dataset example from the docs raises error in Typescript

Predicting: ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 100.00% | 2/2

Completed
file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/node_modules/langsmith/dist/client.js:1245
            throw new Error(`Invalid run type: ${typeof run}`);
                  ^

Error: Invalid run type: undefined
    at Client.evaluateRun (file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/node_modules/langsmith/dist/client.js:1245:19)
    at file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/dist/smith/runner_utils.js:258:89
    at Array.map (<anonymous>)
    at applyEvaluators (file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/dist/smith/runner_utils.js:258:63)
    at runOnDataset (file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/dist/smith/runner_utils.js:389:29)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at <anonymous> (/Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/evals/benchmark.ts:33:16)

Node.js v18.16.0

Here's the script

import { Client, Run, Example } from 'langsmith';
import { runOnDataset } from 'langchain/smith';
import { EvaluationResult } from 'langsmith/evaluation';

const client = new Client();

// Define dataset: these are your test cases
const datasetName = "Test Dataset";

// // Define your evaluator
const exactMatch = async ({ run, example }: { run: Run; example?: Example; }): Promise<EvaluationResult> => {
    return {
        key: 'exact_match',
        score: run.outputs?.output === example?.outputs?.output ? 1 : 0,
    };
};

(async () => { await runOnDataset(
    (input: { postfix: string }) => ({ output: `Welcome ${input.postfix}` }), // Your AI system goes here
    datasetName, // The data to predict and grade over
    {
        evaluationConfig: { customEvaluators: [exactMatch] },
        projectMetadata: {
            version: "1.0.0",
            revision_id: "beta",
        },
    }
)})();

Suggestion:

No response

hinthornw commented 5 months ago

Trying to reproduce - apologies for the inconvenience