JS `AnswerRelevancy` bug with model configuration

mongodben commented 3 months ago

When using the AnswerRelevancy evaluator from the autoevals npm package, I run into the following error when passing the OpenAI API key to the AnswerRelevancy.openAiApiKey property:

AggregateError: Found exceptions for the following scorers: makeAnswerRelevance
    at callback (/Users/ben.p/projects/chatbot/node_modules/braintrust/dist/cli.js:5113:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/ben.p/projects/chatbot/node_modules/braintrust/dist/cli.js:5136:16 {
  [errors]: [
    BadRequestError: 400 Error: No API keys found (for null). You can configure API secrets at https://www.braintrust.dev/app/settings?subroute=secrets
        at APIError.generate (/Users/ben.p/projects/chatbot/node_modules/autoevals/node_modules/openai/error.js:45:20)
        at OpenAI.makeStatusError (/Users/ben.p/projects/chatbot/node_modules/autoevals/node_modules/openai/core.js:263:33)
        at OpenAI.makeRequest (/Users/ben.p/projects/chatbot/node_modules/autoevals/node_modules/openai/core.js:306:30)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
      status: 400,
      headers: [Object],
      request_id: undefined,
      error: undefined,
      code: undefined,
      param: undefined,
      type: undefined
    }
  ]
}

I do not hav this error with the Faithfulness and ContextRelevancy evaluators.

Here is my source code: https://github.com/mongodb/chatbot/pull/450/files#diff-720d5a77593bb16d732d787d35a130f12ade9b93af9dd912d9a4706657dd6555R31

ankrgyl commented 3 months ago

I've confirmed this reproduces and have a fix

ankrgyl commented 3 months ago

@mongodben we'll publish this in the next release of autoevals (0.0.81) within the next few hours

braintrustdata / autoevals

JS `AnswerRelevancy` bug with model configuration #81