llamacpp example - ValueError: only one element tensors can be converted to Python scalars

[Rubyer77]

Describe the issue as clearly as possible:

I run examples/llamacpp_example.py

outlines/models/llamacpp.py:180: FutureWarning: The input object of type 'Tensor' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'Tensor', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`.
  token_ids = np.array(token_ids)

 ValueError: only one element tensors can be converted to Python scalars

output of poetry show:

accelerate                0.26.1     Accelerate
annotated-types           0.6.0      Reusable constraint types to use with typing.Annotated
asttokens                 2.4.1      Annotate AST trees with source code positions
attrs                     23.2.0     Classes Without Boilerplate
beartype                  0.16.4     Unbearably fast runtime type checking in pure Python.
certifi                   2023.11.17 Python package for providing Mozilla's CA Bundle.
charset-normalizer        3.3.2      The Real First Universal Charset Detector. Open, modern and actively maintained alternative...
cloudpickle               3.0.0      Pickler class to extend the standard pickle.Pickler functionality
diskcache                 5.6.3      Disk Cache -- Disk and file backed persistent cache.
filelock                  3.13.1     A platform independent file lock.
fsspec                    2023.12.2  File-system specification
huggingface-hub           0.20.2     Client library to download and publish models, datasets and other repos on the huggingface....
icontract                 2.6.6      Provide design-by-contract with informative violation messages.
idna                      3.6        Internationalized Domain Names in Applications (IDNA)
interegular               0.3.3      a regex intersection checker
jinja2                    3.1.3      A very fast and expressive template engine.
joblib                    1.3.2      Lightweight pipelining with Python functions
jsonschema                4.20.0     An implementation of JSON Schema validation for Python
jsonschema-specifications 2023.12.1  The JSON Schema meta-schemas and vocabularies, exposed as a Registry
lark                      1.1.9      a modern parsing library
llama-cpp-python          0.2.28     Python bindings for the llama.cpp library
llvmlite                  0.41.1     lightweight wrapper around basic LLVM functionality
markupsafe                2.1.3      Safely add untrusted strings to HTML/XML markup.
mpmath                    1.3.0      Python library for arbitrary-precision floating-point arithmetic
nest-asyncio              1.5.8      Patch asyncio to allow nested event loops
networkx                  3.1        Python package for creating and manipulating graphs and networks
numba                     0.58.1     compiling Python code using LLVM
numpy                     1.24.4     Fundamental package for array computing in Python
outlines                  0.0.23     Probabilistic Generative Model Programming
packaging                 23.2       Core utilities for Python packages
perscache                 0.6.1      An easy to use decorator for persistent memoization: like `functools.lrucache`, but results...
psutil                    5.9.7      Cross-platform lib for process and system monitoring in Python.
pydantic                  2.5.3      Data validation using Python type hints
pydantic-core             2.14.6
pyyaml                    6.0.1      YAML parser and emitter for Python
referencing               0.32.1     JSON Referencing + Python
regex                     2023.12.25 Alternative regular expression module, to replace re.
requests                  2.31.0     Python HTTP for Humans.
rpds-py                   0.17.1     Python bindings to Rust's persistent data structures (rpds)
safetensors               0.4.1
scipy                     1.9.3      Fundamental algorithms for scientific computing in Python
six                       1.16.0     Python 2 and 3 compatibility utilities
sympy                     1.12       Computer algebra system (CAS) in Python
tokenizers                0.15.0
torch                     2.1.2      Tensors and Dynamic neural networks in Python with strong GPU acceleration
tqdm                      4.66.1     Fast, Extensible Progress Meter
transformers              4.36.2     State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
typing-extensions         4.9.0      Backported and Experimental Type Hints for Python 3.8+
urllib3                   2.1.0      HTTP library with thread-safe connection pooling, file post, and more.

Steps/code to reproduce the bug:

run `examples/llamacpp_example.py`

only thing I changed is the model, I used "./tinyllama-1.1b-chat-v1.0.Q5_K_M.gguf" after downloading it.

Expected result:

running without ValueError

Error message:

No response

Outlines/Python version information:

0.0.23

3.9.6 (default, Jan 16 2022, 16:12:04) 
[Clang 12.0.0 (clang-1200.0.32.29)]

Context for the issue:

No response

[hongjunyan]

I have encountered the same issue. I downloaded phi-2.Q4_K_M.gguf according to the instructions in the documentation, but I get ValueError: only one element tensors can be converted to Python scalars.

[msjgriffiths]

I patched the code to bypass this error (ran into a second). FYI

    def decode(self, token_ids: NDArray[np.int64]) -> List[str]:
+        if isinstance(token_ids, torch.Tensor):
+            token_ids = token_ids.numpy()
         if isinstance(token_ids, list):
             token_ids = np.array(token_ids)
         if token_ids.ndim == 1:
             token_ids = [token_ids]

[Rubyer77]

I patched the code to bypass this error (ran into a second). FYI

    def decode(self, token_ids: NDArray[np.int64]) -> List[str]:
+        if isinstance(token_ids, torch.Tensor):
+            token_ids = token_ids.numpy()
         if isinstance(token_ids, list):
             token_ids = np.array(token_ids)
         if token_ids.ndim == 1:
             token_ids = [token_ids]

error is still present for me with that patch

[Rubyer77]

https://github.com/outlines-dev/outlines/pull/548 seems to help 🤗