hadjebi commented 1 year ago

The bug A lot of example notebooks don't work properly

To Reproduce

Just running the sample notebooks. A lot of them don't work properly. Am I missing something?

As an example, notebook guaranteeing_valid_syntax.ipynb doesn't work. When I run this cell:

we can pre-define valid option sets

valid_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]

define the prompt

program = guidance("""The following is a character profile for an RPG game in JSON format.


{
    "description": "{{description}}",
    "name": "{{gen 'name'}}",
    "age": {{gen 'age' pattern='[0-9]+' stop=','}},
    "armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
    "weapon": "{{select 'weapon' options=valid_weapons}}",
    "class": "{{gen 'class'}}",
    "mantra": "{{gen 'mantra'}}",
    "strength": {{gen 'strength' pattern='[0-9]+' stop=','}},
    "items": [{{#geneach 'items' num_iterations=3}}
        "{{gen 'this'}}",{{/geneach}}
    ]
}```""")

# execute the prompt
out = program(description="A quick and nimble fighter.", valid_weapons=valid_weapons)
-------------------------------------------------------------------------------------------------------

It starts generating some output and then stops and just outputs the following: 
=============================================================
The following is a character profile for an RPG game in JSON format.
```json
{
    "description": "{{description}}",
    "name": "{{gen 'name'}}",
    "age": {{gen 'age' pattern='[0-9]+' stop=','}},
    "armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
    "weapon": "{{select 'weapon' options=valid_weapons}}",
    "class": "{{gen 'class'}}",
    "mantra": "{{gen 'mantra'}}",
    "strength": {{gen 'strength' pattern='[0-9]+' stop=','}},
    "items": [{{#geneach 'items' num_iterations=3}}
        "{{gen 'this'}}",{{/geneach}}
    ]
}```
=============================================================

I am aware that {{or}} is not working in some cases, so changed it to: 
-------------------------------------------------------------------------------------------------------
# we can pre-define valid option sets
valid_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]
valid_armors = ["leather", "chainmail", "plate"]
# define the prompt
program = guidance("""The following is a character profile for an RPG game in JSON format.
```json
{
    "description": "{{description}}",
    "name": "{{gen 'name'}}",
    "age": {{gen 'age' pattern='[0-9]+' stop=','}},
    "armor": "{{select 'armor' options=valid_armors}}",
    "weapon": "{{select 'weapon' options=valid_weapons}}",
    "class": "{{gen 'class'}}",
    "mantra": "{{gen 'mantra'}}",
    "strength": {{gen 'strength' pattern='[0-9]+' stop=','}},
    "items": [{{#geneach 'items' num_iterations=3}}
        "{{gen 'this'}}",{{/geneach}}
    ]
}```""")

# execute the prompt
out = program(
    description="A quick and nimble fighter.", 
    valid_armors=valid_armors,
    valid_weapons=valid_weapons)
-------------------------------------------------------------------------------------------------------

But it still doesn't work. The strange thing is, it starts outputting some fields which are not even part of the json schema (like stats, inventory, etc.)

**System info (please complete the following information):**
 - OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Linux  5.10.0-24-cloud-amd64 #1 SMP Debian 5.10.179-5 (2023-08-08) x86_64 GNU/Linux

 - Guidance Version (`guidance.__version__`):  guidance  0.0.64

❯ pip list
Package                       Version
----------------------------- ------------
accelerate                    0.21.0
aiohttp                       3.8.5
aiosignal                     1.3.1
alembic                       1.11.1
altair                        5.0.1
anthropic                     0.3.6
anyio                         3.7.1
appdirs                       1.4.4
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
arrow                         1.2.3
astroid                       2.15.6
asttokens                     2.0.5
async-lru                     2.0.4
async-timeout                 4.0.2
asyncio                       3.4.3
asyncpg                       0.28.0
attrs                         23.1.0
Babel                         2.12.1
backcall                      0.2.0
backoff                       2.2.1
beautifulsoup4                4.12.2
bleach                        6.0.0
blinker                       1.6.2
blis                          0.7.10
boilerpy3                     1.0.6
boltons                       23.0.0
cachetools                    5.3.1
canals                        0.2.2
catalogue                     2.0.9
cattrs                        23.1.2
certifi                       2023.5.7
cffi                          1.15.1
chardet                       5.1.0
charset-normalizer            3.2.0
chroma-hnswlib                0.7.1
chromadb                      0.4.2
click                         8.1.6
cloud-sql-python-connector    1.3.0
cloudpickle                   2.2.1
cmake                         3.27.0
cohere                        4.17.0
colorama                      0.4.6
colored                       2.2.3
coloredlogs                   15.0.1
comm                          0.1.3
confection                    0.1.0
config                        0.5.1
contourpy                     1.1.0
cryptography                  41.0.2
ctranslate2                   3.17.1
cycler                        0.11.0
cymem                         2.0.7
databricks-cli                0.17.7
dataclasses-json              0.5.13
datasets                      2.13.1
debugpy                       1.6.7
decorator                     5.1.1
defusedxml                    0.7.1
dill                          0.3.6
diskcache                     5.6.1
distro                        1.8.0
dnspython                     2.4.0
docker                        6.1.3
docopt                        0.6.2
duckdb                        0.8.1
elastic-transport             8.4.0
elasticsearch                 8.9.0
eliot                         1.14.0
eliot-tree                    21.0.0
en-core-web-sm                3.6.0
entrypoints                   0.4
et-xmlfile                    1.1.0
Events                        0.4
executing                     0.8.3
faiss-cpu                     1.7.4
Faker                         19.2.0
farm-haystack                 1.18.1
fastapi                       0.100.1
fastjsonschema                2.18.0
favicon                       0.7.0
filelock                      3.12.2
filetype                      1.2.0
Flask                         2.3.2
flatbuffers                   23.5.26
fonttools                     4.41.1
fqdn                          1.5.1
frozendict                    2.3.8
frozenlist                    1.4.0
fsspec                        2023.6.0
gitdb                         4.0.10
GitPython                     3.1.32
google-ai-generativelanguage  0.2.0
google-api-core               2.11.1
google-api-python-client      2.94.0
google-auth                   2.22.0
google-auth-httplib2          0.1.0
google-cloud-aiplatform       1.28.1
google-cloud-bigquery         3.11.4
google-cloud-core             2.3.3
google-cloud-resource-manager 1.10.2
google-cloud-storage          2.10.0
google-crc32c                 1.5.0
google-generativeai           0.1.0
google-resumable-media        2.5.0
google-search-results         2.4.2
googleapis-common-protos      1.59.1
gptcache                      0.1.38
greenlet                      2.0.2
griffe                        0.32.3
grpc-google-iam-v1            0.12.6
grpcio                        1.56.2
grpcio-status                 1.56.2
guardrails-ai                 0.1.8
guidance                      0.0.64
gunicorn                      20.1.0
h11                           0.14.0
haystack-entailment-checker   0.0.4
hf-hub-ctranslate2            2.12.0
htbuilder                     0.6.1
httpcore                      0.17.3
httplib2                      0.22.0
httptools                     0.6.0
httpx                         0.24.1
huggingface-hub               0.16.4
humanfriendly                 10.0
idna                          3.4
importlib-metadata            6.8.0
importlib-resources           6.0.0
inflect                       7.0.0
inspiredco                    0.0.2
ipykernel                     6.24.0
ipython                       8.14.0
ipywidgets                    8.0.7
iso8601                       2.0.0
isoduration                   20.11.0
isort                         5.12.0
itsdangerous                  2.1.2
jedi                          0.18.2
Jinja2                        3.1.2
jmespath                      1.0.1
joblib                        1.3.1
json5                         0.9.14
jsonpointer                   2.4
jsonschema                    4.18.4
jsonschema-specifications     2023.7.1
jupyter_client                8.3.0
jupyter_core                  4.12.0
jupyter-events                0.7.0
jupyter-lsp                   2.2.0
jupyter_server                2.7.0
jupyter_server_terminals      0.4.4
jupyterlab                    4.0.3
jupyterlab-pygments           0.2.2
jupyterlab_server             2.24.0
jupyterlab-widgets            3.0.8
kaggle                        1.5.16
kiwisolver                    1.4.4
lanarky                       0.7.12
langchain                     0.0.250
langcodes                     3.3.0
langsmith                     0.0.12
lark                          1.1.7
lazy-imports                  0.3.1
lazy-object-proxy             1.9.0
lit                           16.0.6
llama-index                   0.7.11.post1
loguru                        0.7.0
lxml                          4.9.3
Mako                          1.2.4
Markdown                      3.4.3
markdown-it-py                3.0.0
markdownlit                   0.0.7
MarkupSafe                    2.1.3
marshmallow                   3.20.1
matplotlib                    3.7.2
matplotlib-inline             0.1.2
mccabe                        0.7.0
mdurl                         0.1.2
merkle-json                   1.0.0
millify                       0.1.1
mistune                       3.0.1
mlflow                        2.5.0
monotonic                     1.6
more-itertools                9.1.0
mpmath                        1.3.0
msal                          1.23.0
msg-parser                    1.2.0
multidict                     6.0.4
multiprocess                  0.70.14
munch                         4.0.0
murmurhash                    1.0.9
mypy-extensions               1.0.0
nbclient                      0.8.0
nbconvert                     7.7.3
nbformat                      5.9.2
neo4j                         5.11.0
nest-asyncio                  1.5.1
networkx                      3.1
nltk                          3.8.1
notebook_shim                 0.2.3
num2words                     0.5.12
numexpr                       2.8.4
numpy                         1.25.1
nvidia-cublas-cu11            11.10.3.66
nvidia-cuda-cupti-cu11        11.7.101
nvidia-cuda-nvrtc-cu11        11.7.99
nvidia-cuda-runtime-cu11      11.7.99
nvidia-cudnn-cu11             8.5.0.96
nvidia-cufft-cu11             10.9.0.58
nvidia-curand-cu11            10.2.10.91
nvidia-cusolver-cu11          11.4.0.1
nvidia-cusparse-cu11          11.7.4.91
nvidia-nccl-cu11              2.14.3
nvidia-nvtx-cu11              11.7.91
oauthlib                      3.2.2
olefile                       0.46
onnxruntime                   1.15.1
openai                        0.27.8
openapi-schema-pydantic       1.2.4
openpyxl                      3.1.2
overrides                     7.3.1
packaging                     21.3
pandas                        2.0.3
pandocfilters                 1.5.0
parso                         0.8.3
pathy                         0.10.2
pdf2image                     1.16.3
pdfminer.six                  20221105
pexpect                       4.8.0
pgvector                      0.2.1
pickleshare                   0.7.5
Pillow                        9.5.0
pinecone-client               2.2.2
pip                           23.2
platformdirs                  3.9.1
posthog                       3.0.1
preshed                       3.0.8
prometheus-client             0.17.1
prompt-toolkit                3.0.39
prompthub-py                  4.0.0
prompttools                   0.0.24
proto-plus                    1.22.3
protobuf                      4.23.4
psutil                        5.9.5
ptyprocess                    0.7.0
pulsar-client                 3.2.0
pure-eval                     0.2.2
pyarrow                       12.0.1
pyasn1                        0.5.0
pyasn1-modules                0.3.0
pycparser                     2.21
pydantic                      1.10.11
pydeck                        0.8.0
Pygments                      2.15.1
pygtrie                       2.5.0
PyJWT                         2.8.0
pylint                        2.17.5
pymdown-extensions            10.1
Pympler                       1.0.1
pypandoc                      1.11
pyparsing                     3.0.4
pypdf                         3.12.2
pypdfium2                     4.18.0
pyphen                        0.14.0
PyPika                        0.48.9
pyre-extensions               0.0.29
pyrsistent                    0.19.3
python-dateutil               2.8.2
python-decouple               3.8
python-docx                   0.8.11
python-dotenv                 1.0.0
python-json-logger            2.0.7
python-magic                  0.4.27
python-pptx                   0.6.21
python-slugify                8.0.1
pytz                          2023.3
pytz-deprecation-shim         0.1.0.post0
PyYAML                        6.0.1
pyzmq                         25.1.0
quantulum3                    0.9.0
querystring-parser            1.2.4
rank-bm25                     0.2.2
referencing                   0.30.0
regex                         2023.6.3
replicate                     0.9.0
requests                      2.31.0
requests-cache                0.9.8
rfc3339-validator             0.1.4
rfc3986-validator             0.1.1
rich                          13.4.2
rpds-py                       0.9.2
rsa                           4.9
safetensors                   0.3.1
scikit-learn                  1.3.0
scipy                         1.11.1
Send2Trash                    1.8.2
sentence-transformers         2.2.2
sentencepiece                 0.1.99
setuptools                    68.0.0
Shapely                       1.8.5.post1
six                           1.16.0
slack-bolt                    1.18.0
slack-sdk                     3.21.3
smart-open                    6.3.0
smmap                         5.0.0
sniffio                       1.3.0
soupsieve                     2.4.1
spacy                         3.6.0
spacy-legacy                  3.0.12
spacy-loggers                 1.0.4
SQLAlchemy                    2.0.19
sqlparse                      0.4.4
srsly                         2.4.7
sseclient-py                  1.7.2
st-annotated-text             4.0.0
stack-data                    0.2.0
starlette                     0.27.0
streamlit                     1.25.0
streamlit-aggrid              0.3.4.post3
streamlit-camera-input-live   0.2.0
streamlit-card                0.0.61
streamlit-embedcode           0.1.2
streamlit-extras              0.2.7
streamlit-faker               0.0.2
streamlit-image-coordinates   0.1.6
streamlit-javascript          0.1.5
streamlit-keyup               0.2.0
streamlit-toggle-switch       1.0.2
streamlit-vertical-slider     1.0.2
sympy                         1.12
tabulate                      0.9.0
tenacity                      8.2.2
terminado                     0.17.1
text-unidecode                1.3
textstat                      0.7.3
thinc                         8.1.10
threadpoolctl                 3.2.0
tiktoken                      0.4.0
tinycss2                      1.2.1
tokenizers                    0.13.3
toml                          0.10.2
tomlkit                       0.12.1
toolz                         0.12.0
torch                         2.0.1
torchvision                   0.15.2
tornado                       6.3.2
tqdm                          4.65.0
traitlets                     5.9.0
transformers                  4.30.1
triton                        2.0.0
trulens-eval                  0.6.0
typer                         0.9.0
typing_extensions             4.5.0
typing-inspect                0.8.0
tzdata                        2023.3
tzlocal                       4.3.1
unstructured                  0.8.1
uri-template                  1.3.0
uritemplate                   4.1.1
url-normalize                 1.4.3
urllib3                       1.26.15
uvicorn                       0.23.1
uvloop                        0.17.0
validators                    0.20.0
wasabi                        1.1.2
watchdog                      3.0.0
watchfiles                    0.19.0
wcwidth                       0.2.5
webcolors                     1.13
webencodings                  0.5.1
websocket-client              1.6.1
websockets                    11.0.3
Werkzeug                      2.3.6
wheel                         0.40.0
widgetsnbextension            4.0.8
wikipedia                     1.4.0
wrapt                         1.15.0
xformers                      0.0.20
xlrd                          2.0.1
XlsxWriter                    3.1.2
xxhash                        3.2.0
yarl                          1.9.2
zipp                          3.16.2
zope.interface                6.0

yudataguy commented 1 year ago

same issue. set the api key, still getting template as output

SamMakesThings commented 1 year ago

Guidance seems to output the template whenever there's some internal error with generation - doesn't seem to pass any errors through to the user to fix. E.g.

Outputs template if

API key is missing
Usage cap is hit
Trying to use features like select with chat models

Really just seems like pretty unusable error handling 😢 Wanted to use Guidance for a project but now that seems like a bad idea

yudataguy commented 1 year ago

I'm going to try https://github.com/ShreyaR/guardrails for output parser

Harsha-Nori commented 1 year ago

Hi, I think this is all related to errors not being surfaced in specific environments. As a short term fix, whenever the template gets returned unexecuted, you can inspect the program._exception property to see if there are any errors.

Related thread where we're specifically discussing this issue: https://github.com/guidance-ai/guidance/issues/353

hadjebi commented 1 year ago

I have tried Guardrails. The idea behind Guidance is nicer but looks like it's not compatible with the new OpenAI models.

lawwu commented 1 year ago

Facing similar errors. program._exception reports:

AssertionError("The OpenAI API does not support Guidance pattern controls! Please either switch to an endpoint that does, or don't use the `pattern` argument to `gen`.")

chrsevs commented 1 year ago

I've also got a similar issue. In VSCode, trying to run the very first example of ~gen in tutorial.ipynb, the template is returned.

Mac OS Ventura 13.5.1 Python 3.9.6 Guidance 0.0.64

I'm using an Azure OpenAI model that I've set up as instructed here. Everything runs without throwing errors, program._exception is empty and program.log reports: No exception occured.

Archer-Thane commented 1 year ago

I have this error: AssertionError('When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting.') my code is :

import guidance
guidance.llm = guidance.llms.OpenAI("gpt-3.5-turbo", api_key="somekey")
options = ['Yes', 'No', 'Maybe']

program = guidance('''Is the following sentence offensive?
Sentence: {{example}}
Answer: {{select "answer" options=options}}''')

executed_program = program(example='I hate your tacos', options=options)
executed_program._exception

ynishi commented 1 year ago

I have got same error, it maybe because of current default open ai model is chat model.

you can change model like this, it will work. https://github.com/guidance-ai/guidance#rich-output-structure-example-notebook

# set the default language model used to execute guidance programs
guidance.llm = guidance.llms.OpenAI("text-davinci-003")

PiotrCzapla commented 1 year ago

it looks like this is caused by gen not being able to figure out the proper stop word anymore, see #360

yotahassan commented 12 months ago

Hey! thought this would be helpful.

Ran into the same issue as everyone here. The solution was to add the api key. Attaching a screenshot to show what to add.

If you don't have an API key already, get one from this link: https://platform.openai.com/account/api-keys

marcotcr commented 11 months ago

Hey, sorry, we had a lot of bugs everywhere. Hopefully this one is fixed in the new release. Sorry it took us so long to get to it!

guidance-ai / guidance

A lot of example notebooks don't work properly #344

we can pre-define valid option sets

define the prompt