Response from GPT4 is sometimes invalid JSON

[jplindstrom]

It seems that sometimes GPT-4 doesn't return valid JSON, using """ as some kind of string delimiter:

      "fileContents": """
import os
import json
... etc ...
"""

Does it need a stricter prompt?

Here's a full failed run:

promptr my-script.py -p "fix all the FIX comments in this Python script"
Execution time: 23615ms
There was an error parsing the model's output:
{
  "operations": [
    {
      "crudOperation": "update",
      "filePath": "my-script.py",
      "fileContents": """
import os
import json
...
"""
    }
  ]
}
Attemping to extract json:
{
  "operations": [
    {
      "crudOperation": "update",
      "filePath": "my-script.py",
      "fileContents": """
import os
import json
...
"""
    }
  ]
}
SyntaxError: Unexpected string in JSON at position 147
    at JSON.parse (<anonymous>)
    at extractOperationsFromOutput (file:///home/myuser/.nvm/versions/node/v16.19.0/lib/node_modules/@ifnotnowwhen/promptr/src/services/ExtractOperationsFromOutput.js:14:29)
    at Function.call (file:///home/myuser/.nvm/versions/node/v16.19.0/lib/node_modules/@ifnotnowwhen/promptr/src/services/PromptrService.js:57:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Function.call (file:///home/myuser/.nvm/versions/node/v16.19.0/lib/node_modules/@ifnotnowwhen/promptr/src/Main.js:38:12)
    at async file:///home/myuser/.nvm/versions/node/v16.19.0/lib/node_modules/@ifnotnowwhen/promptr/bin/index.js:4:18
file:///home/myuser/.nvm/versions/node/v16.19.0/lib/node_modules/@ifnotnowwhen/promptr/src/services/RefactorResultProcessor.js:6
    data.operations.forEach((operation) => {
         ^

TypeError: Cannot read properties of null (reading 'operations')
    at Function.call (file:///home/myuser/.nvm/versions/node/v16.19.0/lib/node_modules/@ifnotnowwhen/promptr/src/services/RefactorResultProcessor.js:6:10)
    at Function.call (file:///home/myuser/.nvm/versions/node/v16.19.0/lib/node_modules/@ifnotnowwhen/promptr/src/services/PromptrService.js:62:37)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Function.call (file:///home/myuser/.nvm/versions/node/v16.19.0/lib/node_modules/@ifnotnowwhen/promptr/src/Main.js:38:12)
    at async file:///home/myuser/.nvm/versions/node/v16.19.0/lib/node_modules/@ifnotnowwhen/promptr/bin/index.js:4:18

[ferrislucas]

Hi @jplindstrom thanks for opening an issue. Are you using the most recent version? If not, could you update to the latest and try again?

[jplindstrom]

Yeah I just did a normal npm install so that would be the latest I guess.

promptr --version
6.0.0

[jplindstrom]

This doesn't happen 100% of the time. I tried a small elisp program, and that worked.

It also happens with GPT-3.5

[ferrislucas]

Thanks again for reporting the issue. Do you happen to have a public example that I can use to reproduce the problem?

I haven’t seen this problem since version 3.0.6, but I’m working mostly in ruby and JavaScript. I’ll try messing with some python projects and see if I can get it to happen.

[ferrislucas]

Hi @jplindstrom , I just released a fix for this. I hope it resolves the problem that you’re seeing. Please reopen this issue if you see it happen again.