search

dottxt-ai / outlines

Structured Text Generation
https://dottxt-ai.github.io/outlines/
Apache License 2.0
9.57k stars 490 forks source link

RuntimeError when using generate.json() on llama 3.2 with llamaccp #1261

Closed adder closed 1 week ago

adder commented 1 week ago

Describe the issue as clearly as possible:

When trying to use generate.json with llama 3.2 and llamaccp I get a runtime error: RuntimeError: Cannot convert token ` �` (30433) to bytes: �"

Steps/code to reproduce the bug:

from outlines import models, generate
from pydantic import BaseModel, Field
model = models.llamacpp(
    "bartowski/Llama-3.2-1B-Instruct-GGUF", "Llama-3.2-1B-Instruct-Q4_K_M.gguf"
)

class Sections(BaseModel):
    sections: str
generator = generate.json(model, Sections)

Expected result:

No error

Error message:

{
    "name": "RuntimeError",
    "message": "Cannot convert token ` �` (30433) to bytes:  �",
    "stack": "---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[102], line 9
      7 class Sections(BaseModel):
      8     sections: str
----> 9 generator = generate.json(model, Sections)

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/functools.py:907, in singledispatch.<locals>.wrapper(*args, **kw)
    903 if not args:
    904     raise TypeError(f'{funcname} requires at least '
    905                     '1 positional argument')
--> 907 return dispatch(args[0].__class__)(*args, **kw)

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/site-packages/outlines/generate/json.py:49, in json(model, schema_object, sampler, whitespace_pattern)
     47     schema = pyjson.dumps(schema_object.model_json_schema())
     48     regex_str = build_regex_from_schema(schema, whitespace_pattern)
---> 49     generator = regex(model, regex_str, sampler)
     50     generator.format_sequence = lambda x: schema_object.parse_raw(x)
     51 elif callable(schema_object):

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/functools.py:907, in singledispatch.<locals>.wrapper(*args, **kw)
    903 if not args:
    904     raise TypeError(f'{funcname} requires at least '
    905                     '1 positional argument')
--> 907 return dispatch(args[0].__class__)(*args, **kw)

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/site-packages/outlines/generate/regex.py:61, in regex_llamacpp(model, regex_str, sampler)
     53 @regex.register(LlamaCpp)
     54 def regex_llamacpp(
     55     model: LlamaCpp,
     56     regex_str: str,
     57     sampler: Sampler = multinomial(),
     58 ):
     59     from outlines.integrations.llamacpp import RegexLogitsProcessor
---> 61     logits_processor = RegexLogitsProcessor(regex_str, llm=model.model)
     62     return SequenceGeneratorAdapter(model, logits_processor, sampler)

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/site-packages/outlines/integrations/llamacpp.py:129, in RegexLogitsProcessor.__init__(self, regex_string, llm)
    119 \"\"\"Compile the FSM that drives the regex-guided generation.
    120 
    121 Parameters
   (...)
    126     The Llama model.
    127 \"\"\"
    128 tokenizer = LlamaCppTokenizer(model=llm)
--> 129 fsm = RegexGuide(regex_string, tokenizer)
    130 super().__init__(tokenizer=tokenizer, fsm=fsm)

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/site-packages/outlines/fsm/guide.py:145, in RegexGuide.__init__(self, regex_string, tokenizer)
    140 def __init__(self, regex_string: str, tokenizer):
    141     (
    142         self.states_to_token_maps,
    143         self.empty_token_ids,
    144         fsm_finals,
--> 145     ) = create_states_mapping(regex_string, tokenizer)
    146     self.eos_token_id = tokenizer.eos_token_id
    147     self.final_states = fsm_finals | {-1}

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/site-packages/outlines/caching.py:122, in cache.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    119 result = wrapper.__memory__.get(cache_key, default=ENOVAL, retry=True)
    121 if result is ENOVAL:
--> 122     result = cached_function(*args, **kwargs)
    123     wrapper.__memory__.set(cache_key, result, expire, retry=True)
    125 return result

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/site-packages/outlines/fsm/guide.py:118, in create_states_mapping(regex_string, tokenizer)
    116 byte_fsm = make_byte_level_fsm(regex_pattern.to_fsm().reduce(), keep_utf8=True)
    117 regex_fsm, _ = make_deterministic_fsm(byte_fsm)
--> 118 states_to_token_maps, empty_token_ids = create_fsm_index_tokenizer(
    119     regex_fsm, tokenizer
    120 )
    122 # We make sure that it is possible to generate strings in the language
    123 # of the regular expression with the tokens present in the model's
    124 # vocabulary.
    125 if not any(
    126     regex_fsm.finals.intersection(v.values()) for v in states_to_token_maps.values()
    127 ):

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/site-packages/outlines/fsm/regex.py:898, in create_fsm_index_tokenizer(fsm, tokenizer)
    885 def create_fsm_index_tokenizer(
    886     fsm: BetterFSM,
    887     tokenizer: \"Tokenizer\",
    888 ) -> Tuple[Dict[int, Dict[int, int]], Set[int]]:
    889     \"\"\"Construct an FMS index from a tokenizer.
    890 
    891     This uses the end-to-end approach of `create_fsm_index_end_to_end`.
   (...)
    896 
    897     \"\"\"
--> 898     vocabulary, empty_token_ids = reduced_vocabulary(tokenizer)
    900     states_to_token_subsets = create_fsm_index_end_to_end(fsm.fsm_info, vocabulary)
    902     # Allow transitions to EOS from all terminals FSM states that are
    903     # reachable
    904     # TODO: Do we really need this anymore?

File ~/micromamba/envs/sanofi-immunogenicity-nlp-llm/lib/python3.12/site-packages/outlines/fsm/regex.py:861, in reduced_vocabulary(tokenizer)
    857         token_bytes = cast(
    858             List[int], [gpt2_unicode_to_bytes().get(c) for c in token]
    859         )
    860         if None in token_bytes:
--> 861             raise RuntimeError(
    862                 f\"Cannot convert token `{token}` ({token_idx}) to bytes: {token_str}\"
    863             )
    864     token_str = \"\".join(byte_symbol(b) for b in token_bytes)
    866 vocabulary.setdefault(token_str, []).append(token_idx)

RuntimeError: Cannot convert token ` �` (30433) to bytes:  �"
}

Outlines/Python version information:

Version information 0.0.46

0.0.46
Python 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0]
accelerate @ file:///home/conda/feedstock_root/build_artifacts/accelerate_1728744679987/work
aiohappyeyeballs @ file:///home/conda/feedstock_root/build_artifacts/aiohappyeyeballs_1727779797566/work
aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1728629003519/work
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
aiostream @ file:///home/conda/feedstock_root/build_artifacts/aiostream_1698703327740/work
annotated-types @ file:///home/conda/feedstock_root/build_artifacts/annotated-types_1716290248287/work
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1728935693959/work
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1698341106958/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1722977137225/work
beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1705564648255/work
black @ file:///home/conda/feedstock_root/build_artifacts/black-recipe_1728503730172/work
Brotli @ file:///home/conda/feedstock_root/build_artifacts/brotli-split_1725267488082/work
certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1725278078093/work/certifi
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1725560558132/work
cfgv @ file:///home/conda/feedstock_root/build_artifacts/cfgv_1629909281805/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1728479282467/work
click @ file:///home/conda/feedstock_root/build_artifacts/click_1692311806742/work
cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1729059237860/work
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1666700638685/work
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1710320294760/work
coverage @ file:///home/conda/feedstock_root/build_artifacts/coverage_1729610035939/work
dataclasses-json @ file:///home/conda/feedstock_root/build_artifacts/dataclasses-json_1717969336599/work
datasets @ file:///home/conda/feedstock_root/build_artifacts/datasets_1729718892862/work
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1728594098955/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
deepsearch-glm==0.26.1
Deprecated @ file:///home/conda/feedstock_root/build_artifacts/deprecated_1685233314779/work
dill @ file:///home/conda/feedstock_root/build_artifacts/dill_1706434688412/work
dirtyjson @ file:///home/conda/feedstock_root/build_artifacts/dirtyjson_1706872027011/work
diskcache @ file:///home/conda/feedstock_root/build_artifacts/diskcache_1693471346238/work
distlib @ file:///home/conda/feedstock_root/build_artifacts/distlib_1728557174656/work
distro @ file:///home/conda/feedstock_root/build_artifacts/distro_1704321475663/work
dnspython @ file:///home/conda/feedstock_root/build_artifacts/dnspython_1728178720319/work
docling==2.4.0
docling-core==2.3.1
docling-ibm-models==2.0.3
docling-parse==2.0.2
docutils==0.21.2
easyocr==1.7.2
email_validator @ file:///home/conda/feedstock_root/build_artifacts/email-validator-meta_1718984626658/work
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1720869315914/work
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1725214404607/work
fastapi @ file:///home/conda/feedstock_root/build_artifacts/fastapi_1730122729340/work
fastapi-cli @ file:///home/conda/feedstock_root/build_artifacts/fastapi-cli_1728947708687/work
filelock @ file:///home/conda/feedstock_root/build_artifacts/filelock_1726613473834/work
filetype==1.2.0
frozenlist @ file:///home/conda/feedstock_root/build_artifacts/frozenlist_1729699457945/work
fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1725543257300/work
gguf @ file:///home/conda/feedstock_root/build_artifacts/bld/rattler-build_gguf_1727807692/work
gmpy2 @ file:///home/conda/feedstock_root/build_artifacts/gmpy2_1725379832114/work
greenlet @ file:///home/conda/feedstock_root/build_artifacts/greenlet_1726922184666/work
h11 @ file:///home/conda/feedstock_root/build_artifacts/h11_1664132893548/work
h2 @ file:///home/conda/feedstock_root/build_artifacts/h2_1634280454336/work
hpack==4.0.0
httpcore @ file:///home/conda/feedstock_root/build_artifacts/httpcore_1727820890233/work
httptools @ file:///home/conda/feedstock_root/build_artifacts/httptools_1726688090923/work
httpx @ file:///home/conda/feedstock_root/build_artifacts/httpx_1724778349782/work
huggingface_hub @ file:///home/conda/feedstock_root/build_artifacts/huggingface_hub_1730211885663/work
hyperframe @ file:///home/conda/feedstock_root/build_artifacts/hyperframe_1619110129307/work
identify @ file:///home/conda/feedstock_root/build_artifacts/identify_1726369553067/work
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1726459485162/work
imageio==2.36.0
importlib_metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1726082825846/work
importlib_resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1725921340658/work
interegular @ file:///home/conda/feedstock_root/build_artifacts/interegular_1704610508103/work
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1719845459717/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1729866374957/work
ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1724334859652/work
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1715127149914/work
jiter @ file:///home/conda/feedstock_root/build_artifacts/jiter_1728396196396/work
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1714665484399/work
jsonlines==3.1.0
jsonref==1.1.0
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema_1720529478715/work
jsonschema-specifications @ file:///tmp/tmpvslgxhz5/src
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1726610684920/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1727163409502/work
jupyterlab_widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1724331334887/work
lark @ file:///home/conda/feedstock_root/build_artifacts/lark_1723661507334/work
lazy_loader==0.4
llama-cloud @ file:///home/conda/feedstock_root/build_artifacts/llama-cloud_1729721602089/work
llama-index @ file:///home/conda/feedstock_root/build_artifacts/llama-index_1729846257249/work
llama-index-agent-openai @ file:///home/conda/feedstock_root/build_artifacts/llama-index-agent-openai_1726799640930/work
llama-index-cli @ file:///home/conda/feedstock_root/build_artifacts/llama-index-cli_1725944866394/work
llama-index-core @ file:///home/conda/feedstock_root/build_artifacts/llama-index-core_1729830144973/work
llama-index-embeddings-openai @ file:///home/conda/feedstock_root/build_artifacts/llama-index-embeddings-openai_1726288106632/work
llama-index-indices-managed-llama-cloud @ file:///home/conda/feedstock_root/build_artifacts/llama-index-indices-managed-llama-cloud_1726351640307/work
llama-index-legacy @ file:///home/conda/feedstock_root/build_artifacts/llama-index-legacy_1724015056889/work
llama-index-llms-openai @ file:///home/conda/feedstock_root/build_artifacts/llama-index-llms-openai_1729649363917/work
llama-index-multi-modal-llms-openai @ file:///home/conda/feedstock_root/build_artifacts/llama-index-multi-modal-llms-openai_1729662219565/work
llama-index-program-openai @ file:///home/conda/feedstock_root/build_artifacts/llama-index-program-openai_1725603575491/work
llama-index-question-gen-openai @ file:///home/conda/feedstock_root/build_artifacts/llama-index-question-gen-openai_1725603544671/work
llama-index-readers-file @ file:///home/conda/feedstock_root/build_artifacts/llama-index-readers-file_1726780421715/work
llama-index-readers-llama-parse @ file:///home/conda/feedstock_root/build_artifacts/llama-index-readers-llama-parse_1725327478184/work
llama-parse @ file:///home/conda/feedstock_root/build_artifacts/llama-parse_1729896442653/work
llama_cpp_python==0.3.1
llamaindex-py-client @ file:///home/conda/feedstock_root/build_artifacts/llamaindex-py-client_1714162494295/work
llvmlite==0.43.0
lxml==4.9.4
markdown-it-py @ file:///home/conda/feedstock_root/build_artifacts/markdown-it-py_1686175045316/work
marko==2.1.2
MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1729351299924/work
marshmallow @ file:///home/conda/feedstock_root/build_artifacts/marshmallow_1729200873190/work
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1713250518406/work
mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1704317613764/work
mean-average-precision==2021.4.26.0
mpmath @ file:///home/conda/feedstock_root/build_artifacts/mpmath_1678228039184/work
multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1729065475820/work
multiprocess @ file:///home/conda/feedstock_root/build_artifacts/multiprocess_1724954620400/work
mypy-extensions @ file:///home/conda/feedstock_root/build_artifacts/mypy_extensions_1675543315189/work
nest_asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1705850609492/work
networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1729530778722/work
ninja==1.11.1.1
nltk @ file:///home/conda/feedstock_root/build_artifacts/nltk_1724022741962/work
nodeenv @ file:///home/conda/feedstock_root/build_artifacts/nodeenv_1717585263558/work
numba @ file:///home/conda/feedstock_root/build_artifacts/numba_1718888016245/work
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1707225359967/work/dist/numpy-1.26.4-cp312-cp312-linux_x86_64.whl#sha256=031b7d6b2e5e604d9e21fc21be713ebf28ce133ec872dce6de006742d5e49bab
openai @ file:///home/conda/feedstock_root/build_artifacts/openai_1730324042552/work
opencv-python-headless==4.10.0.84
outlines @ file:///home/conda/feedstock_root/build_artifacts/outlines_1719136138681/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1718189413536/work
pandas @ file:///home/conda/feedstock_root/build_artifacts/pandas_1726878417179/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1712320355065/work
pathspec @ file:///home/conda/feedstock_root/build_artifacts/pathspec_1702249949303/work
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1706113125309/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
pillow==10.4.0
pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1694617248815/work
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1726613481435/work
pre_commit @ file:///home/conda/feedstock_root/build_artifacts/pre-commit_1728420161621/work
prompt_toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1727341649933/work
propcache @ file:///home/conda/feedstock_root/build_artifacts/propcache_1728545779122/work
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1729847057810/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure_eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1721585709575/work
pyairports @ file:///home/conda/feedstock_root/build_artifacts/pyairports_1717577825901/work
pyarrow==16.1.0
pyclipper==1.3.0.post6
pycountry @ file:///home/conda/feedstock_root/build_artifacts/pycountry_1718094481023/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1711811537435/work
pydantic @ file:///home/conda/feedstock_root/build_artifacts/pydantic_1726601062926/work
pydantic-settings @ file:///home/conda/feedstock_root/build_artifacts/pydantic-settings_1729717917977/work
pydantic_core @ file:///home/conda/feedstock_root/build_artifacts/pydantic-core_1726524971346/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1714846767233/work
PyMuPDF==1.24.13
pymupdf4llm==0.0.17
pypdf @ file:///home/conda/feedstock_root/build_artifacts/pypdf-split_1721653426016/work/dist
pypdfium2==4.30.0
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
python-bidi==0.6.3
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1709299778482/work
python-docx==1.1.2
python-dotenv @ file:///home/conda/feedstock_root/build_artifacts/python-dotenv-split_1706018097647/work
python-multipart @ file:///home/conda/feedstock_root/build_artifacts/python-multipart_1730103284353/work
python-pptx==1.0.2
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1706886791323/work
PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1725456134219/work
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1728642254015/work
referencing @ file:///home/conda/feedstock_root/build_artifacts/referencing_1714619483868/work
regex @ file:///home/conda/feedstock_root/build_artifacts/regex_1726095591219/work
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1717057054362/work
rich @ file:///home/conda/feedstock_root/build_artifacts/rich_1729622917073/work/dist
rpds-py @ file:///home/conda/feedstock_root/build_artifacts/rpds-py_1725327050544/work
Rtree==1.3.0
safetensors @ file:///home/conda/feedstock_root/build_artifacts/safetensors_1725631981923/work
scikit-image==0.24.0
scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy-split_1729480682811/work/dist/scipy-1.14.1-cp312-cp312-linux_x86_64.whl#sha256=5dc72baa2d67ff6e6fc7aa7dd0fe21390ed1e3400bdeddd731d9d88066ffbd26
sentencepiece @ file:///home/conda/feedstock_root/build_artifacts/sentencepiece-split_1709763329758/work/python
setuptools==75.1.0
shapely==2.0.6
shellingham @ file:///home/conda/feedstock_root/build_artifacts/shellingham_1698144360966/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1708952932303/work
soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1693929250441/work
SQLAlchemy @ file:///home/conda/feedstock_root/build_artifacts/sqlalchemy_1729066352888/work
sse-starlette @ file:///home/conda/feedstock_root/build_artifacts/sse-starlette_1722520017366/work
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
starlette @ file:///home/conda/feedstock_root/build_artifacts/starlette_1730226025142/work
starlette-context @ file:///home/conda/feedstock_root/build_artifacts/starlette-context_1676743447281/work
striprtf @ file:///home/conda/feedstock_root/build_artifacts/striprtf_1711715151288/work
sympy==1.13.1
tabulate==0.9.0
tenacity @ file:///home/conda/feedstock_root/build_artifacts/tenacity_1720351771156/work
tifffile==2024.9.20
tiktoken @ file:///home/conda/feedstock_root/build_artifacts/tiktoken_1728083954548/work
tokenizers @ file:///home/conda/feedstock_root/build_artifacts/tokenizers_1728588649775/work/bindings/python
tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1727974628237/work
torch==2.5.1+cpu
torchvision==0.20.1+cpu
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1724956131631/work
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1730145186354/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1713535121073/work
transformers @ file:///home/conda/feedstock_root/build_artifacts/transformers_1730244979226/work
typer==0.12.5
typer-slim==0.12.5
types-futures @ file:///home/conda/feedstock_root/build_artifacts/types-futures_1643318201097/work
types-protobuf @ file:///home/conda/feedstock_root/build_artifacts/types-protobuf_1713343418376/work
typing-inspect @ file:///home/conda/feedstock_root/build_artifacts/typing_inspect_1685820062773/work
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1717802530399/work
tzdata @ file:///home/conda/feedstock_root/build_artifacts/python-tzdata_1727140567071/work
ukkonen @ file:///home/conda/feedstock_root/build_artifacts/ukkonen_1725784043978/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1726496430923/work
uvicorn @ file:///home/conda/feedstock_root/build_artifacts/uvicorn_1730219695027/work
uvloop @ file:///home/conda/feedstock_root/build_artifacts/uvloop_1730214319645/work
virtualenv @ file:///home/conda/feedstock_root/build_artifacts/virtualenv_1730204812392/work
watchfiles @ file:///home/conda/feedstock_root/build_artifacts/watchfiles_1725346965467/work
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1704731205417/work
websockets @ file:///home/conda/feedstock_root/build_artifacts/websockets_1727013341365/work
wheel==0.44.0
widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1724331337528/work
wrapt @ file:///home/conda/feedstock_root/build_artifacts/wrapt_1724957898456/work
XlsxWriter==3.2.0
xxhash @ file:///home/conda/feedstock_root/build_artifacts/python-xxhash_1725272030893/work
yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1729798396041/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1726248574750/work
zstandard==0.23.0

Context for the issue:

No response

adder commented 1 week ago

I noticed that condaforge was still on version 0.0.46. I updated with pip to the latest version 0.1.13 I still get the same error.

cpfiffer commented 1 week ago

Can you try manually setting the tokenizer like they do here?

import llama_cpp
from outlines import generate, models

model = models.llamacpp(
  "bartowski/Llama-3.2-1B-Instruct-GGUF", 
  "Llama-3.2-1B-Instruct-Q4_K_M.gguf",
  tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained(
    "meta-llama/Llama-3.2-1B"
   ),
)
adder commented 1 week ago

Thanks That is the solution. I acutally used instead

model = models.llamacpp(
    "bartowski/Llama-3.2-1B-Instruct-GGUF", "Llama-3.2-1B-Instruct-Q4_K_M.gguf"
    ## tokenizer needs to be set manually
, tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained(
            "unsloth/Llama-3.2-1B-Instruct"
            ))

Since you need to use credentials to get things from the meta-llama repo

cpfiffer commented 1 week ago

Perfect. Closing for now, glad it worked!