Sometimes `fnCall.arguments` returns a string so `JSON.parse()` fails

[programmarchy]

I noticed with this prompt:

wz resize hero.jpg to a 180x180 thumbnail

Sometimes I would get an exception:

SyntaxError: Unexpected token c in JSON at position 0

But sometimes the command would succeed.

Added some logging, and appears that sometimes fnCall.arguments is a string instead of an object { description, content }.

Here's my log:

λ wz resize hero.jpg to a 180x180 thumbnail
✔ whiz is thinking...
fnCall.arguments = {
  "description": "Resize hero.jpg to a 180x180 thumbnail",
  "content": "convert hero.jpg -resize 180x180 thumbnail.jpg"
}
✔ Resize hero.jpg to a 180x180 thumbnail

Are you sure you want to run: convert hero.jpg -resize 180x180 thumbnail.jpg? (y/N) · false
Aborting...
λ wz resize hero.jpg to a 180x180 thumbnail
✔ whiz is thinking...
fnCall.arguments = convert hero.jpg -resize 180x180 thumbnail.jpg
SyntaxError: Unexpected token c in JSON at position 0
    at JSON.parse (<anonymous>)
    at main (file:///Users/programmarchy/Library/pnpm/global/5/.pnpm/whiz_cli@0.0.6-alpha.0/node_modules/whiz_cli/build/index.js:38:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[searchableguy]

Yeah, it's inconsistent when you use the default turbo model but it's also the cheapest to use (so far, it costs me < $0.5 for testing and personal use)

Switching to gpt4 makes it way more consistent.

I'm thinking of ways to modify prompt or add more validation to re-prompt if it is incorrect. Feel free to open a PR with ideas to make it more consistent and reliable.

I will improve the error message in the meantime.

[programmarchy]

This is hacky, but would you be opposed to something like:

function parseArgs(fnCall, context: { content }) {
  if (typeof fnCall.arguments === 'string') {
    return { description: content, content: fnCall.arguments };
  }  else {
    return JSON.parse(fnCall.arguments);
  }
}

[searchableguy]

I would rather fail than prompt for a command which doesn't work or has any possibility of causing issues while running. The latter is more frustrating than failing fast.

So, I don't think that would work. I think we should focus on improving reliability or allowing quick iterations.

[programmarchy]

Agree with what you said, but the interesting thing is the command was correct (convert hero.jpg -resize 180x180 thumbnail.jpg), it just wasn't formatted according to the output spec. My guess is that ChatGPT knew the command to run, but not how to describe it, so it broke the output spec to return only the command. I suppose there's not a way to know that for sure, though. Perhaps we could give it more leeway in the description field by allowing null or adding something like "if you cannot describe the command, return an empty string" with the hope that it follows the output spec more closely.

[searchableguy]

That sounds like a good idea. Wanna open a PR with some tests to the functions file to improve the output?

[searchableguy]

Someone made a pull request to improve this. Would you mind running a couple test on your end?

[searchableguy]

So I tested it with your commands - it seems to work more reliably so closing this issue for now. Feel free to re-open if you find examples that do not work.