JS: err running braintrust CLI

[mongodben]

when i run the braintrust CLI, i get the following error.

full trace:

$ npx braintrust eval src/verifiedAnswers.eval.ts
▲ [WARNING] "./xhr-sync-worker.js" should be marked as external for use with "require.resolve" [require-resolve-not-external]

    ../../node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:31:57:
      31 │ const syncWorkerFile = require.resolve ? require.resolve("./xhr-sync-worker.js") : null;
         ╵                                                          ~~~~~~~~~~~~~~~~~~~~~~

Failed to compile /Users/ben.p/projects/chatbot/packages/chatbot-server-mongodb-public/src/verifiedAnswers.eval.ts: Cannot find module './xhr-sync-worker.js'
Require stack:
- /Users/ben.p/projects/chatbot/packages/chatbot-server-mongodb-public/node_modules/braintrust/dist/cli.js
Processing 0 evaluators...

using braintrust CLI 0.0.167

here's the source code as well https://github.com/mongodb/chatbot/pull/550/

[ankrgyl]

There's a long tail of complex JS that npx braintrust eval can't bundle. This is probably one such example. In these cases, we recommend running the eval script directly, i.e. npx tsx src/verifiedAnswers.eval.ts, or bundling it yourself and running the generated JS

[mongodben]

i wonder if its related to our recent change to use NodeNext module resolution https://github.com/mongodb/chatbot/blob/main/tsconfig.json#L5-L6

already have it running from pure node script 👍

[mongodben]

i got the braintrust CLI working on these files by removing a dependent module that uses jsdom. seems that the issue relates to the way braintrust does bundling.

i see braintrust CLI has the option braintrust eval --bundle, which is described as:

  --bundle              Experimental (do not use unless you know what you're doing)

i wonder if using this could be a path to resolving the issue w/o removing the dependent module. i didn't see any docs on this flag option, though. if --bundle might be useful here, could you provide some insight on how to use it?

[ankrgyl]

No, --bundle is an (outdated) alias for --push and unrelated.

I think the tl;dr of these bundling issues is that we use esbuild with a set of flags that (for the most part) solves the bundling problem. To support the very long tail, I think we need to add some extensibility, either via additional/custom flags that forward to esbuild, or letting you define the esbuild bundle command (or maybe even another bundle command) yourself.

[mongodben]

To support the very long tail, I think we need to add some extensibility, either via additional/custom flags that forward to esbuild, or letting you define the esbuild bundle command (or maybe even another bundle command) yourself.

this seems reasonable to me.

feel free to close this issue if you want to do a separate one for the esbuild extensibility.