Play around with LLM prompts on ChatGPT

[awissink]

We need to figure out how to best prompt GPT to receive correct output for the node editor. First, let's explore what happens when we put different prompts into ChatGPT and see what we get out. Will put our findings here.

[awissink]

When feeding ChatGPT this prompt, we get out a valid JSON for additive synthesis.

Given this format of a JSON file for AM/FM synthesis in a WebAudio node editor, generate me a JSON for additive synthesis: {
    "nodes": [
        {
            "id": "fd6bbc5f9754162f",
            "name": "Oscillator",
            "data": { "baseFreq": 440, "waveform": "triangle" }
        },
        { "id": "eaf61eeae013a92e", "name": "Gain", "data": { "gain": 0.5 } },
        {
            "id": "e28d78cfb85364af",
            "name": "Universal Output",
            "data": { "gain": 0.8 }
        },
        {
            "id": "79e92d958baff7f5",
            "name": "Oscillator",
            "data": { "baseFreq": 8, "waveform": "triangle" }
        },
        { "id": "63863aafacb20fd9", "name": "Gain", "data": { "gain": 8 } },
        {
            "id": "805296b4c54b1660",
            "name": "Oscillator",
            "data": { "baseFreq": 100, "waveform": "triangle" }
        },
        { "id": "f8572925cd49003e", "name": "Gain", "data": { "gain": 0.5 } }
    ],
    "connections": [
        {
            "source": "eaf61eeae013a92e",
            "sourceOutput": "signal",
            "target": "e28d78cfb85364af",
            "targetInput": "signal"
        },
        {
            "source": "fd6bbc5f9754162f",
            "sourceOutput": "signal",
            "target": "eaf61eeae013a92e",
            "targetInput": "signal"
        },
        {
            "source": "79e92d958baff7f5",
            "sourceOutput": "signal",
            "target": "63863aafacb20fd9",
            "targetInput": "signal"
        },
        {
            "source": "63863aafacb20fd9",
            "sourceOutput": "signal",
            "target": "fd6bbc5f9754162f",
            "targetInput": "frequency"
        },
        {
            "source": "805296b4c54b1660",
            "sourceOutput": "signal",
            "target": "f8572925cd49003e",
            "targetInput": "signal"
        },
        {
            "source": "f8572925cd49003e",
            "sourceOutput": "signal",
            "target": "eaf61eeae013a92e",
            "targetInput": "additionalGain"
        }
    ]
}

So it seems like, for a simplistic case, GPT can generate a configuration from scratch.

[awissink]

Tried to ask it to add another node to an existing setup and it timed out.

Given this JSON of WebAudio nodes in a node editor, add another sawtooth oscillator node connection at frequency 440: {"nodes":[{"id":"fd6bbc5f9754162f","name":"Oscillator","data":{"baseFreq":440,"waveform":"triangle"}},{"id":"eaf61eeae013a92e","name":"Gain","data":{"gain":0.5}},{"id":"e28d78cfb85364af","name":"Universal Output","data":{"gain":0.8}},{"id":"79e92d958baff7f5","name":"Oscillator","data":{"baseFreq":8,"waveform":"triangle"}},{"id":"63863aafacb20fd9","name":"Gain","data":{"gain":8}},{"id":"805296b4c54b1660","name":"Oscillator","data":{"baseFreq":100,"waveform":"triangle"}},{"id":"f8572925cd49003e","name":"Gain","data":{"gain":0.5}}],"connections":[{"source":"eaf61eeae013a92e","sourceOutput":"signal","target":"e28d78cfb85364af","targetInput":"signal"},{"source":"fd6bbc5f9754162f","sourceOutput":"signal","target":"eaf61eeae013a92e","targetInput":"signal"},{"source":"79e92d958baff7f5","sourceOutput":"signal","target":"63863aafacb20fd9","targetInput":"signal"},{"source":"63863aafacb20fd9","sourceOutput":"signal","target":"fd6bbc5f9754162f","targetInput":"frequency"},{"source":"805296b4c54b1660","sourceOutput":"signal","target":"f8572925cd49003e","targetInput":"signal"},{"source":"f8572925cd49003e","sourceOutput":"signal","target":"eaf61eeae013a92e","targetInput":"additionalGain"}]}

Will try re-running and see what the output is.

[awissink]

Ok seems like that was just bad wifi on my part. Ran the same prompt again and got JSON output which I then plugged into the node editor:

It works, but interesting how it connects two things to the same input point.

[awissink]

Let's try removing the node we just added.

Given this JSON of WebAudio nodes in a node editor, remove the sawtooth oscillator node. {
    "nodes": [
        {
            "id": "fd6bbc5f9754162f",
            "name": "Oscillator",
            "data": {
                "baseFreq": 440,
                "waveform": "triangle"
            }
        },
        {
            "id": "eaf61eeae013a92e",
            "name": "Gain",
            "data": {
                "gain": 0.5
            }
        },
        {
            "id": "e28d78cfb85364af",
            "name": "Universal Output",
            "data": {
                "gain": 0.8
            }
        },
        {
            "id": "79e92d958baff7f5",
            "name": "Oscillator",
            "data": {
                "baseFreq": 8,
                "waveform": "triangle"
            }
        },
        {
            "id": "63863aafacb20fd9",
            "name": "Gain",
            "data": {
                "gain": 8
            }
        },
        {
            "id": "805296b4c54b1660",
            "name": "Oscillator",
            "data": {
                "baseFreq": 100,
                "waveform": "triangle"
            }
        },
        {
            "id": "f8572925cd49003e",
            "name": "Gain",
            "data": {
                "gain": 0.5
            }
        },
        {
            "id": "new_oscillator_sawtooth_440",
            "name": "Oscillator",
            "data": {
                "baseFreq": 440,
                "waveform": "sawtooth"
            }
        }
    ],
    "connections": [
        {
            "source": "eaf61eeae013a92e",
            "sourceOutput": "signal",
            "target": "e28d78cfb85364af",
            "targetInput": "signal"
        },
        {
            "source": "fd6bbc5f9754162f",
            "sourceOutput": "signal",
            "target": "eaf61eeae013a92e",
            "targetInput": "signal"
        },
        {
            "source": "79e92d958baff7f5",
            "sourceOutput": "signal",
            "target": "63863aafacb20fd9",
            "targetInput": "signal"
        },
        {
            "source": "63863aafacb20fd9",
            "sourceOutput": "signal",
            "target": "fd6bbc5f9754162f",
            "targetInput": "frequency"
        },
        {
            "source": "805296b4c54b1660",
            "sourceOutput": "signal",
            "target": "f8572925cd49003e",
            "targetInput": "signal"
        },
        {
            "source": "f8572925cd49003e",
            "sourceOutput": "signal",
            "target": "eaf61eeae013a92e",
            "targetInput": "additionalGain"
        },
        {
            "source": "new_oscillator_sawtooth_440",
            "sourceOutput": "signal",
            "target": "eaf61eeae013a92e",
            "targetInput": "signal"
        }
    ]
}

Nice!!

[awissink]

Now, what happens if we're ambiguous? What if we just describe a sound we want?

Given this format of a JSON file for AM/FM synthesis in a WebAudio node editor, generate me a JSON for a sound that sounds like the ocean [insert valid JSON]

Hmmm, it's maybe not as great at this. It returns this JSON:

{
  "nodes": [
    {
      "id": "1",
      "name": "WhiteNoiseGenerator",
      "data": {}
    },
    {
      "id": "2",
      "name": "Oscillator",
      "data": { "baseFreq": 0.5, "waveform": "sine" }
    },
    {
      "id": "3",
      "name": "Gain",
      "data": { "gain": 0.2 }
    },
    {
      "id": "4",
      "name": "LowPassFilter",
      "data": { "frequency": 1000 }
    },
    {
      "id": "5",
      "name": "Gain",
      "data": { "gain": 1 }
    },
    {
      "id": "6",
      "name": "Universal Output",
      "data": {}
    }
  ],
  "connections": [
    {
      "source": "1",
      "sourceOutput": "signal",
      "target": "4",
      "targetInput": "signal"
    },
    {
      "source": "4",
      "sourceOutput": "signal",
      "target": "5",
      "targetInput": "signal"
    },
    {
      "source": "2",
      "sourceOutput": "signal",
      "target": "3",
      "targetInput": "signal"
    },
    {
      "source": "3",
      "sourceOutput": "signal",
      "target": "5",
      "targetInput": "additionalGain"
    },
    {
      "source": "5",
      "sourceOutput": "signal",
      "target": "6",
      "targetInput": "signal"
    }
  ]
}

which doesn't successfully import... looks like ChatGPT hallucinated a "WhiteNoiseGenerator."

[awissink]

When I said it couldn't use that node, here's what it spit out:

{
  "nodes": [
    {
      "id": "noiseOsc1",
      "name": "Oscillator",
      "data": { "baseFreq": 440, "waveform": "triangle" }
    },
    {
      "id": "noiseGain1",
      "name": "Gain",
      "data": { "gain": 0.3 }
    },
    {
      "id": "lfoForNoise",
      "name": "Oscillator",
      "data": { "baseFreq": 0.1, "waveform": "sine" }
    },
    {
      "id": "lfoGain",
      "name": "Gain",
      "data": { "gain": 0.25 }
    },
    {
      "id": "output",
      "name": "Universal Output",
      "data": { "gain": 0.8 }
    }
  ],
  "connections": [
    {
      "source": "noiseOsc1",
      "sourceOutput": "signal",
      "target": "noiseGain1",
      "targetInput": "signal"
    },
    {
      "source": "noiseGain1",
      "sourceOutput": "signal",
      "target": "output",
      "targetInput": "signal"
    },
    {
      "source": "lfoForNoise",
      "sourceOutput": "signal",
      "target": "lfoGain",
      "targetInput": "signal"
    },
    {
      "source": "lfoGain",
      "sourceOutput": "signal",
      "target": "noiseGain1",
      "targetInput": "additionalGain"
    }
  ]
}

That works! But it doesn't really sound like the ocean...

Seems like in order to be successful, GPT needs an intimate knowledge of what's allowed versus what isn't. We need to give it a well-defined set of nodes it's allowed to use as well as examples.

[awissink]

Seems like when I gave it a one shot example, GPT thought that I wanted it to be related to AM/FM synthesis, even though I didn't necessarily care. It really seems like we are leaning towards needing a larger knowledge base of examples here.

[santolucito]

add enhancement label since this is pretty open ended. probably need to split this into more concrete subtasks as issues. I will tackle this myself later.