examples: guidance - Githubissues

mudler commented 1 year ago

Would be fantastic to have a full working example on how to run guidance: https://github.com/microsoft/guidance

A first try works, however:

we need to consolidate the work on stopwords (#64), but llama-cpp models and rwkv should work now already
find a good model that fits with the example

I've been trying with:

import guidance
import os

# set the default language model used to execute guidance programs
guidance.llm = guidance.llms.OpenAI(model=os.environ.get('OPENAI_API_MODEL', 'text-davinci-003'), caching=False)

# define a guidance program that adapts a proverb
program = guidance("""Tweak this proverb to apply to model instructions instead.

{{proverb}}
- {{book}} {{chapter}}:{{verse}}

UPDATED
Where there is no guidance{{gen 'rewrite' stop="\\n-"}}
- GPT {{gen 'chapter'}}:{{gen 'verse'}}""")

# execute the program on a specific proverb
executed_program = program(
    proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.",
    book="Proverbs",
    chapter=11,
    verse=14
)

print(executed_program)
print(guidance('The link is <a href="http:{{gen max_tokens=10}}')())

ralyodio commented 1 year ago

what is guidance?

Wittionary commented 1 year ago

@ralyodio

Guidance enables you to control modern language models more effectively and efficiently than traditional prompting or chaining. Guidance programs allow you to interleave generation, prompting, and logical control into a single continuous flow matching how the language model actually processes the text.

Source: microsoft/guidance

thistleknot commented 1 year ago

using the same snippet of code, I get

UPDATED Where there is no guidance Traceback (most recent call last): File "C:\Users\User\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program_executor.py", line 94, in run await self.visit(self.parse_tree) File "C:\Users\User\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program_executor.py", line 428, in visit visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node)) File "C:\Users\User\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program_executor.py", line 428, in visit visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node)) File "C:\Users\User\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program_executor.py", line 395, in visit command_output = await command_function(*positional_args, **named_args) File "C:\Users\User\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance\library_select.py", line 158, in select option_logprobs = await recursive_select("") File "C:\Users\User\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance\library_select.py", line 115, in recursive_select logprobs_result = gen_obj["choices"][0]["logprobs"] KeyError: 'logprobs'

Error in program: 'logprobs'

KeyError Traceback (most recent call last) Cell In[7], line 17 7 program = guidance("""Tweak this proverb to apply to model instructions instead. 8 9 {{proverb}} (...) 13 Where there is no guidance{{gen 'rewrite' stop="\n-"}} 14 - GPT {{#select 'chapter'}}9{{or}}10{{or}}11{{/select}}:{{gen 'verse'}}""") 16 # execute the program on a specific proverb ---> 17 executed_program = program( 18 proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.", 19 book="Proverbs", 20 chapter=11, 21 verse=14 22 )

File ~\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program.py:224, in Program.call(self, **kwargs) 222 loop = asyncio.new_event_loop() 223 loop.create_task(new_program.update_display.run()) # start the display updater --> 224 loop.run_until_complete(new_program.execute()) 226 return new_program

File ~\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\nest_asyncio.py:89, in _patch_loop..run_until_complete(self, future) 86 if not f.done(): 87 raise RuntimeError( 88 'Event loop stopped before Future completed.') ---> 89 return f.result()

File ~\AppData\Local\Programs\Python\Python39\lib\asyncio\futures.py:201, in Future.result(self) 199 self.__log_traceback = False 200 if self._exception is not None: --> 201 raise self._exception 202 return self._result

File ~\AppData\Local\Programs\Python\Python39\lib\asyncio\tasks.py:256, in Task.step(failed resolving arguments) 252 try: 253 if exc is None: 254 # We use the send method directly, because coroutines 255 # don't have `iterandnext__` methods. --> 256 result = coro.send(None) 257 else: 258 result = coro.throw(exc)

File ~\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program.py:310, in Program.execute(self) 308 else: 309 with self.llm.session(asynchronous=True) as llm_session: --> 310 await self._executor.run(llm_session) 311 self._text = self._executor.prefix 313 # delete the executor and so mark the program as not executing

File ~\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program_executor.py:98, in ProgramExecutor.run(self, llm_session) 96 print(traceback.format_exc()) 97 print("Error in program: ", e) ---> 98 raise e

File ~\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program_executor.py:94, in ProgramExecutor.run(self, llm_session) 88 self.llm_session = llm_session 89 try: 90 # first parse all the whitespace control 91 # self.whitespace_control_visit(self.parse_tree) 92 93 # now execute the program ---> 94 await self.visit(self.parse_tree) 95 except Exception as e: 96 print(traceback.format_exc())

File ~\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program_executor.py:428, in ProgramExecutor.visit(self, node, next_node, next_next_node, prev_node, parent_node, grandparent_node) 426 else: 427 inner_prev_node = prev_node --> 428 visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node)) 429 # visited_children = [self.visit(child) for child in node.children] 431 if len(visited_children) == 1:

File ~\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance_program_executor.py:395, in ProgramExecutor.visit(self, node, next_node, next_next_node, prev_node, parent_node, grandparent_node) 393 # call the optionally asyncronous command 394 if inspect.iscoroutinefunction(command_function): --> 395 command_output = await command_function(*positional_args, *named_args) 396 else: 397 command_output = command_function(positional_args, **named_args)

File ~\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance\library_select.py:158, in select(variable_name, options, logprobs, list_append, _parser_context) 155 return logprobs_out 157 # recursively compute the logprobs for each option --> 158 option_logprobs = await recursive_select("") 160 selected_option = max(option_logprobs, key=option_logprobs.get) 162 # see if we are appending to a list or not

File ~\AppData\Local\Programs\3.9-JupyterLab\lib\site-packages\guidance\library_select.py:115, in select..recursive_select(current_prefix, allow_token_extension) 106 # generate the token logprobs 107 gen_obj = await parser.llm_session( 108 call_prefix, # TODO: perhaps we should allow passing of token ids directly? 109 max_tokens=1, (...) 113 token_healing=False # we manage token boundary healing ourselves for this function 114 ) --> 115 logprobs_result = gen_obj["choices"][0]["logprobs"] 116 top_logprobs = logprobs_result["top_logprobs"][0] 118 # for each possible next token, see if it grows the prefix in a valid way

KeyError: 'logprobs'

localai-bot commented 1 year ago

:warning::warning::warning::warning::warning:

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

_but.... I can also be funny or helpful :smilecat: and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me, I don't support (yet) replying!

:warning::warning::warning::warning::warning:

This looks like a good start! However, it seems that the OpenAI model in the example is not available in the current format. To use the new format, you can add the following code to convert the model from the old format to the new format:

import llms
import os
# set the default language model used to execute guidance programs
guidance.llm = llms.OpenAI(model_path=os.environ['OPENAI_MODEL_PATH'], model_name='text-davinci-003', caching=False)
# define a guidance program that adapts a proverb
program = guidance("""Tweak this proverb to apply to model instructions instead.
{{proverb}}
- {{book}} {{chapter}}:{{verse}}

UPDATED
Where there is no guidance{{gen 'rewrite' stop="\\n-"}}
- GPT {{gen 'chapter'}}:{{gen 'verse'}}""")
# execute the program on a specific proverb
executed_program = program(
    proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.",
    book="Proverbs",
    chapter=11,
    verse=14
)
# convert the model to the new format
model_path = os.environ['OPENAI_MODEL_PATH']
model_name = 'text-davinci-003'
new_model_path = llms.ModelPath(model_path, model_name=model_name, version='latest')
guidance.llm = llms.OpenAI(model_path=new_model_path, caching=False)
# print the executed program and the updated link
print(executed_program)
print(guidance('The link is <a href="http:{{gen max_tokens=10}}')())

This should convert the model to the new format and allow you to execute the guidance program as expected.

Sources:

mudler / LocalAI

examples: guidance #304

Error in program: 'logprobs'

:warning::warning::warning::warning::warning:

:warning::warning::warning::warning::warning: