Run narration and code writing in parallel

JuggernautJf commented 3 months ago

currently the format is :

const script = [
    {
        type: 'narrate',
        content:
            "First I'm creating the canvas with a size of 640 by 360 pixels.",
    },
    {
        type: 'code',
        content: ['function setup() {', '    createCanvas(640, 360);', '}'],
    },
    {
        type: 'narrate',
        content:
            "Now I'm creating the draw loop which will run continuously. It will call the branch method to start drawing the tree.",
    },
    {
        type: 'code',
        content: [
            'function draw() {',
            '    background(0);',
            '    stroke(255);',
            '    strokeWeight(2);',
            '    translate(width * 0.5, height);',
            '    branch(100);',
            '}',
        ],
    },
    {
        type: 'narrate',
        content:
            'The branch method is a recursive function that draws the tree. It takes a length parameter and draws a line of that length.',
    },
    ...
]

supercrafter100 commented 3 months ago

The current sequence setup is made like it is because it's made with the idea that the language model would create these kind of sections. The code would then generate such sequence that can be executed by the code. See https://chat.openai.com/share/4ae72a32-4b8f-439e-a7a1-cc4aaf516849

supercrafter100 commented 3 months ago

Additionally, the script.js is temporary just to get a working demo. Once the actual AI model get's implemented the script will get discarded as it will get generated on the fly with the response of the LLM.

JuggernautJf commented 3 months ago

I want to change it to :

/**
 * The script for iteration through AiAgent
 * @type {Array<StepGroup>}
 * @typedef {Step[]} StepGroup group of steps to be done simultanously
 * @typedef {Object} Step single step
 * @property {string} type
 * @property {Array<string>} content
 */
const script = [
    [
        {
            type: 'narrate',
            content:[
                "First I'm creating the canvas with a size of 640 by 360 pixels.",
            ]
        },
        {
            type: 'code',
            content: [
                'function setup() {', '    createCanvas(640, 360);', '}'
            ],
        }
    ],
    [
        {
            type: 'narrate',
            content:[
                "Now I'm creating the draw loop which will run continuously. It will call the branch method to start drawing the tree.",
            ]
        },
        {
            type: 'code',
            content: [
                'function draw() {',
                '    background(0);',
                '    stroke(255);',
                '    strokeWeight(2);',
                '    translate(width * 0.5, height);',
                '    branch(100);',
                '}',
            ],
        }
    ],
    [
        {
            type: 'narrate',
            content:[
                'The branch method is a recursive function that draws the tree. It takes a length parameter and draws a line of that length.'
            ]
        },
        {
            type: 'code',
            content: [
                'function branch(len) {',
                '    line(0, 0, 0, -len);',
                '    translate(0, -len);',
                '    if (len > 4) {',
                '        push();',
                '        rotate(PI/4);',
                '        branch(len * 0.67);',
                '        pop();',
                '        push();',
                '        rotate(-PI/4);',
                '        branch(len * 0.67);',
                '        pop();',
                '    }',
                '}',
            ],
        }
    ]
]

Reason: executing steps to be done simultaneously,

list of StepGroup in script would be executed one after another, in each StepGroup, each Step would be done simultaneously without waiting for step before to complete
⭐ Why do i need that? : Right now it is 💢waiting to complete narration💢 to start writing code. that narration about the code and writing can be done simultaneously this way
Proposed format:
```
type Script = StepGroup[]
type StepGroup = Step[]
type Step = {
type: string,
content: string[]
}
```

supercrafter100 commented 3 months ago

This could cause a lot of complication though in terms of when to actually do something in parallel. If everything is just after eachother it's pretty straightforward extracting that from a response of an LLM. We don't always want to talk at the same time of writing because we aren't certain that the message above the code has anything to do with the code below it. I certainly see the benefit of it but rather sceptical if we can actually manage to get that to work without issues once the AI itself is implemented.

JuggernautJf commented 3 months ago

orrr it can be simplified to just another type of step like step.type = "wait" < when this comes from response, extension would wait till all ongoing steps are completed, allowing parrallel walk and talk ( in this case.. code and talk )

dipamsen commented 3 months ago

The LLM does not know which piece of text will take how long to speak, or how long to code, so it cannot reliably send "wait" commands at the perfect times such that the speech and coding always is in sync