Open willkurt opened 3 months ago
PierreCarceller
commented
3 months ago This fix the problem. (Or at least in my case)
For those who would like to test in advance:
pip install git+https://github.com/lapp0/outlines.git@fix-json --force-reinstall
lapp0
commented
3 months ago Thanks for directing people to that branch!
Still a work in progress, only a subset of the failure cases are handled right now. Happy to hear more json failure reports to help me ensure I address all problems!
PierreCarceller
commented
2 months ago @lapp0 since yesterday I've noticed other errors despite the fix already commited. Whenever I come across an error, I'll post it here.
PierreCarceller
commented
2 months ago Traceback (most recent call last):
File "/mnt/home/pierre/miniconda3/envs/llmdoc/lib/python3.10/site-packages/pydantic/main.py", line 1160, in parse_raw
obj = parse.load_str_bytes(
File "/mnt/home/pierre/miniconda3/envs/llmdoc/lib/python3.10/site-packages/pydantic/deprecated/parse.py", line 49, in load_str_bytes
return json_loads(b) # type: ignore
File "/mnt/home/pierre/miniconda3/envs/llmdoc/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/mnt/home/pierre/miniconda3/envs/llmdoc/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/mnt/home/pierre/miniconda3/envs/llmdoc/lib/python3.10/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 3328 (char 3327)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/home/pierre/llmdoc/outlines_server/outlines_server/DocumentClassifier.py", line 131, in <module>
File "/mnt/home/pierre/llmdoc/outlines_server/outlines_server/DocumentClassifier.py", line 92, in classify
File "/mnt/home/pierre/miniconda3/envs/llmdoc/lib/python3.10/site-packages/outlines/generate/api.py", line 511, in __call__
return self._format(completions)
File "/mnt/home/pierre/miniconda3/envs/llmdoc/lib/python3.10/site-packages/outlines/generate/api.py", line 487, in _format
return self.format_sequence(sequences)
File "/mnt/home/pierre/miniconda3/envs/llmdoc/lib/python3.10/site-packages/outlines/generate/json.py", line 50, in <lambda>
generator.format_sequence = lambda x: schema_object.parse_raw(x)
File "/mnt/home/pierre/miniconda3/envs/llmdoc/lib/python3.10/site-packages/pydantic/main.py", line 1187, in parse_raw
raise pydantic_core.ValidationError.from_exception_data(cls.__name__, [error])
pydantic_core._pydantic_core.ValidationError: 1 validation error for DocumentClassificationResult
__root__
Expecting ',' delimiter: line 1 column 3328 (char 3327) [type=value_error.jsondecode, input_value='{ "label": "tax_notice",...99999999999999999999999', input_type=str]
cpfiffer
commented
2 months ago Anything I can help with here? Currently getting bit by this issue, using the dev branch.
import outlines
import llama_cpp
# Load the model
model = outlines.models.llamacpp(
"NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF",
"Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf",
tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained(
"NousResearch/Hermes-2-Pro-Llama-3-8B"
),
n_gpu_layers=-1,
n_ctx=8192,
verbose=False
)
# Specify temperature in sampler
sampler = outlines.samplers.multinomial(samples=100)
# Define the coinflip choice
coinflip_regex_pattern = r"[H|T]"
generator = outlines.generate.choice(
model,
["H", "T"],
sampler=sampler
)
output = generator("Flip a coin, respond with H or T: ")
print(output)
# Count the occurrences of each outcome
heads_count = output.count("H")
tails_count = output.count("T")
print(f"Heads: {heads_count}, Tails: {tails_count}")Versions:
lapp0
commented
2 months ago @cpfiffer Outlines' llama.cpp integration doesn't support multiple samples at once. Could you try again with samples=1 (or just don't explicitly set the sampler)
generator = outlines.generate.choice(
model,
["H", "T"],
)If there's a problem with generate.choice could you also open a separate issue?
@PierreCarceller Can you please share your schema?
The original issue here seems to be resolved by the fix-json branch, except is re-uses logits processors. A new logits processor must be created for each run. This is a separate bug in generate.json / outlines.processors which should be addressed as part of this issue.
cpfiffer
commented
2 months ago I believe the choice issue is in #1109.
Can't believe I missed the multiple samples error,
@cpfiffer Outlines' llama.cpp integration doesn't support multiple samples at once. Could you try again with samples=1 (or just don't explicitly set the sampler)works.
Describe the issue as clearly as possible:
When using
models.llamacppand creating JSON using a Pydantic model I get an error when generating the first result (see code to reproduce below). I have runt his code usingmodels.transformerswith no issue.The model I'm using in this example is taken directly from the Chain of Thought cook book example, but I have also tried others and had the same issue.
Steps/code to reproduce the bug:
Expected result:
Error message:
Outlines/Python version information:
Version information
Context for the issue:
This issue came up while working on an ODSC workshop covering outlines. I ended up going with
transformersinstead ofllama_cpp.