Add support for interactive mode / instructionGPT? (Or add this workaround in the readme?)

[keldenl]

Context

I'm using alpaca.7b on M1 MacBook Pro 32GB ram. I'm trying to get "ChatGPT" like instructions following interactive mode.

The Issue

I tried passing in interactive: true into the Node.js API but didn't work very well likely due to the following piece of code https://github.com/cocktailpeanut/dalai/blob/56850c71328d75c278fd39246495e10da7f3aff7/index.js#L182-L187

Potential Solution

I've gotten the model to trigger interactive mode by updating it to since llama.cpp has support for it (https://github.com/ggerganov/llama.cpp#interactive-mode)

let chunks = []
for(let key in o) {
   chunks.push(`--${key} ${o[key]}`)
}
if (req.interactive) chunks.push('-i')
chunks.push(`-p "${req.prompt}"`)

This still doesn't work quite well. Any chances we could add support for interactive mode? Or maybe I could get some pointers and I can try to figure it out

Current Workaround

I've worked around this (and hopefully this will be helpful for those stumbling upon this) by following similar instructions for instruction mode with alpaca I found in llama.cpp (https://github.com/ggerganov/llama.cpp#instruction-mode-with-alpaca) with alpaca.cpp default parameters

// Raw Prompt is user input
const prompt = `Below is an instruction that describes a task. Write a response that appropriately completes the request.
Question:${rawPrompt}
Response:`;

// Params i'm passing into dalai request
const defaultParams = {
  model: "alpaca.7B",
  temp: 0.1,
  top_k: 40,
  top_p: 0.95,
  repeat_last_n: 64,
  repeat_penalty: 1.3,
  n_predict: 256,
};

// Using it in the request
dalai.request({
        ...defaultParams,
        prompt,
      },
      (token) => {
          res.write(token); // write each token to the response stream
          console.log(token);
        }
      }
    )
    .then(() => {
      res.end(); // end the response when the token generation is complete
    });

Should we add this in the README? And hopefully w can start a discussion about interactive/instruction mode and getting that implementing straight into the node.js api!

[jay5656]

is this why the responses are so bland and sometimes short? if so i woulld love to see this added then!

[gillopillo]

It would be useful to have instructions on how to execute this workaround. you don't really explain what exact files were changed, and how to run it.

[keldenl]

The workaround doesn't include actually modifying anything in dalai (that didn't work well)

i provided the trick i used in the node.js example in the code block, which just prepended additional prompt before the actual instruction you're trying to ask alpaca to do, that's all!

[keldenl]

Specifically

"Below is an instruction that describes a task. Write a response that appropriately completes the request. Question:${YOUR PROMPT HERE} Response:"

Also i linked back to the resource i found this trick in and it has further instructions!

[gillopillo]

Thank you. The only change is the one in dalai/index.js, i see.

[keldenl]

I actually went ahead and opened a pull request to add prompt templates straight into the web-ui so it's more accessible!

https://github.com/cocktailpeanut/dalai/pull/199 <- check it out

[tibzejoker]

I added the interactive mode in this PR.

213

It is not perfect but at least you can experiment with it

[GitGudGandalf]

@tibzejoker your fork is amazing, running on windows 10, im getting it to do some coding! in javascript! unfortunately, I seem to encounter an error where X gets added randomly in my input and outputs. I give you an example.

In chat mode i can use 1 word prompts no problem, but if i add a second word it gets interpreted as a parameter and it fails with this error

dalaialpaca> > [System.Console]::OutputEncoding=[System.Console]::InputEncoding=[System.Text.Encoding]::UTF8; C:UsersXXXXdalaialpacauildReleasemain --seed -1 --threads 30 --n_predict 200 --model models/30B/ggml-model-q4_0.bin --top_k 420 --top_p 0.9 --temp 0.9 --repeat_last_n 64 --repeat_penalty 1.3 -p "The expected response for a highly intelligent chatbot to "write a-i pong game in javascript" is `"" error: unknown argument: a-iX usage: C:UsersXXXXdalaialpacauildReleasemain.exe [options]X X options:X -h, --help show this help message and exitX -i, --interactive run in interactive modeX --interactive-start run in interactive mode and poll user input at startupX -r PROMPT, --reverse-prompt PROMPTX in interactive mode, poll user input upon seeing PROMPTX --color colorise output to distinguish prompt and user input from generations -s SEED, --seed SEED RNG seed (default: -1)X -t N, --threads N number of threads to use during computation (default: 30)X -p PROMPT, --prompt PROMPTX prompt to start generation with (default: random)X -f FNAME, --file FNAMEX prompt file to start generation.X -n N, --n_predict N number of tokens to predict (default: 200)X --top_k N top-k sampling (default: 420)X --top_p N top-p sampling (default: 0.9)X --repeat_last_n N last n tokens to consider for penalize (default: 64)X --repeat_penalty N penalize repeat sequence of tokens (default: 1.3)X -c N, --ctx_size N size of the prompt context (default: 2048)X --temp N temperature (default: 0.9)X -b N, --batch_size N batch size for prompt processing (default: 8)X -m FNAME, --model FNAME

                    model path (default: models/30B/ggml-model-q4_0.bin)

PS C:UsersXXXXdalaialpaca> [System.Console]::OutputEncoding=[System.Console]::InputEncoding=[System.Text.Encoding]::UTF8; C:UsersXXXXdalaialpacauildReleasemain --seed -1 --threads 30 --n_predict 200 --model models/30B/ggml-model-q4_0.bin --top_k 420 --top_p 0.9 --temp 0.9 --repeat_last_n 64 --repeat_penalty 1.3 -p "The expected response for a highly intelligent chatbot to "write a-i pong game in javascript" is `"" error: unknown argument: a-i

usage: C:UsersXXXXdalaialpacauildReleasemain.exe [options]

options: -h, --help show this help message and exit -i, --interactive run in interactive mode --interactive-start run in interactive mode and poll user input at startup -r PROMPT, --reverse-prompt PROMPT in interactive mode, poll user input upon seeing PROMPT --color colorise output to distinguish prompt and user input from generations -s SEED, --seed SEED RNG seed (default: -1) -t N, --threads N number of threads to use during computation (default: 30) -p PROMPT, --prompt PROMPT prompt to start generation with (default: random) -f FNAME, --file FNAME prompt file to start generation. -n N, --n_predict N number of tokens to predict (default: 200) --top_k N top-k sampling (default: 420) --top_p N top-p sampling (default: 0.9) --repeat_last_n N last n tokens to consider for penalize (default: 64) --repeat_penalty N penalize repeat sequence of tokens (default: 1.3) -c N, --ctx_size N size of the prompt context (default: 2048) --temp N temperature (default: 0.9) -b N, --batch_size N batch size for prompt processing (default: 8) -m FNAME, --model FNAME model path (default: models/30B/ggml-model-q4_0.bin) PS C:UsersXXXXdalaialpaca> exit

As you can see there are many X added at the end.

If i run it in normal mode i can use prompts, but i still get random X letters after a few lines generated: example:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

Instruction:

write a pong game in javascript

Response:

// JavaScript Pong Game

import * as p5 from 'p5js'; // import library

let canvas;X // declare variable for DOM element -canvas

const width = window.innerWidth /2, height=window.innertwidth/10 ; // dimensions of game field

let xPosition, yPosistion ,xSpeed,ySpeed;X // ball position and speed variables

// ---initialize-- const playerX=375,playerY=468 ;X // initial values for X axis

canvas = createCanvas(width, height);X // creates canvas object using window.innerWidth/2 (as width) and Window innerHeight/10 as Height

// ---draw setup-- background("#EEE");X // sets the backgound color to a light gray

let rightBoundary=canvas.parentElement.offsetWidth -width; // Sets boundary limit for ball on X axis

const leftBoundary=-width ;X // setting values of boundaries (X and Y)

// ---draw setup-- player1 = createPaddle(0, 625); // creates object with Paddel's coordinate

player2=createPaddle(-87.5,-304 );X // sets the coordinates for the second paddle

//---controls initialization-----------------

let keysPressed;XX //declaring variables

initControls(); let ballVelocity = 16,leftScore ,rightscore ;X // defines velocity of a Ball (this variable is not changed in function)

if (!keysPressed.up){ playerY=player2 .y; }XX // Up key pressed

else if(!keysPressed.down){ ballVelocity=-16, xSpeed =50 ; // Down Key Pressed

} else {xPosition+=ballvelocity ; }X //Moving Ball to left (X axis)

// ---controls initialization----------------- // Up key pressed

if(!keysPressed.left){ yPosistion=player1 .y; }XX // Left Key Pressed

else if (! keysPressed.right ){ xSpeed = -50 ; ballVelocity=-ballvelocity}

// ---controls initialization----------------- else {xPosition+=bv; yPosistion+=(up/down) } // Right Key Pressed

else if (keysPressed.right){ xSpeed = 50 ; ballvelocity= -16;}XX // Down key pressed

// ---controls initialization----------------- else {xPosition+=bv; yPosistion+=(up/down) } // Left Key Pressed