Evaluator on dataset example from the docs raises error in Typescript
Predicting: ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 100.00% | 2/2
Completed
file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/node_modules/langsmith/dist/client.js:1245
throw new Error(`Invalid run type: ${typeof run}`);
^
Error: Invalid run type: undefined
at Client.evaluateRun (file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/node_modules/langsmith/dist/client.js:1245:19)
at file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/dist/smith/runner_utils.js:258:89
at Array.map (<anonymous>)
at applyEvaluators (file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/dist/smith/runner_utils.js:258:63)
at runOnDataset (file:///Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/node_modules/langchain/dist/smith/runner_utils.js:389:29)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at <anonymous> (/Users/RichesofGod/Sienna/VoxzAI/Voxz-Production/evals/benchmark.ts:33:16)
Node.js v18.16.0
Here's the script
import { Client, Run, Example } from 'langsmith';
import { runOnDataset } from 'langchain/smith';
import { EvaluationResult } from 'langsmith/evaluation';
const client = new Client();
// Define dataset: these are your test cases
const datasetName = "Test Dataset";
// // Define your evaluator
const exactMatch = async ({ run, example }: { run: Run; example?: Example; }): Promise<EvaluationResult> => {
return {
key: 'exact_match',
score: run.outputs?.output === example?.outputs?.output ? 1 : 0,
};
};
(async () => { await runOnDataset(
(input: { postfix: string }) => ({ output: `Welcome ${input.postfix}` }), // Your AI system goes here
datasetName, // The data to predict and grade over
{
evaluationConfig: { customEvaluators: [exactMatch] },
projectMetadata: {
version: "1.0.0",
revision_id: "beta",
},
}
)})();
Issue you'd like to raise.
Evaluator on dataset example from the docs raises error in Typescript
Here's the script
Suggestion:
No response