Issue running with Phi3-vision

Describe the issue as clearly as possible:

Should work with Phi3-vision, but running into errors

Steps/code to reproduce the bug:

from outlines import models
from outlines import generate

model = models.transformers("microsoft/Phi-3-vision-128k-instruct", model_kwargs={'trust_remote_code':True, 'device_map':device, '_attn_implementation':'eager'})

schema = """
{
  "title": "User",
  "type": "object",
  "properties": {
    "FirstName": {"type": "string"},
    "LastName": {"type": "string"},
    "ID": {"type": "integer"}
  },
  "required": ["FirstName", "LastName", "LastName"]
}
"""
generator = generate.text(model, schema)
generator(
    "'<|user|>\n<|image_1|>\nExtract detailed JSON<|end|>\n<|assistant|>\n",
    [Image.open(impath)]
)

Expected result:

FirstName: "John"
LastName: "Doe"
ID: "12345678"

Error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[170], line 2
      1 generator = generate.text(model)
----> 2 generator(
      3     "'<|user|>\n<|image_1|>\nExtract detailed JSON<|end|>\n<|assistant|>\n",
      4     [Image.open(impath)]
      5     
      6 )

File ~/miniconda3/envs/all_models/lib/python3.8/site-packages/outlines/generate/api.py:213, in SequenceGenerator.__call__(self, prompts, max_tokens, stop_at, rng)
    209 token_ids = last_state.token_ids
    210 generated_token_ids = self.get_generated_token_ids(
    211     prompt_token_ids, token_ids
    212 )
--> 213 if max_tokens and len(generated_token_ids[0]) >= max_tokens:
    214     break
    215 if stop_sequences and self.is_stop_sequence_found(
    216     self.tokenizer.decode(generated_token_ids), stop_sequences
    217 ):

TypeError: '>=' not supported between instances of 'int' and 'list'

Outlines/Python version information:

Version information

``` huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) 0.0.46 Python 3.8.19 (default, Mar 20 2024, 19:58:24) [GCC 11.2.0] huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using `tokenizers` before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) absl-py==2.1.0 accelerate==0.31.0 aiohttp==3.9.5 aiosignal==1.3.1 anls==0.0.2 annotated-types==0.7.0 anyio @ file:///croot/anyio_1706220167567/work argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work argon2-cffi-bindings @ file:///tmp/build/80754af9/argon2-cffi-bindings_1644569684262/work asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work async-lru @ file:///croot/async-lru_1699554519285/work async-timeout==4.0.3 attrs @ file:///croot/attrs_1695717823297/work Babel @ file:///croot/babel_1671781930836/work backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work beautifulsoup4 @ file:///croot/beautifulsoup4-split_1718029820055/work bitsandbytes==0.42.0 bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work bleu==0.3 Brotli @ file:///croot/brotli-split_1714483155106/work certifi @ file:///croot/certifi_1717618050233/work/certifi cffi @ file:///croot/cffi_1714483155441/work charset-normalizer==3.1.0 click==8.1.7 cloudpickle==3.0.0 comm @ file:///croot/comm_1709322850197/work contourpy==1.1.1 cycler==0.12.1 datasets==2.20.0 debugpy @ file:///croot/debugpy_1690905042057/work decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work dill==0.3.8 dirtyjson==1.0.8 diskcache==5.6.3 docx==0.2.4 efficiency==2.0 einops==0.8.0 et-xmlfile==1.1.0 evaluate==0.1.2 exceptiongroup @ file:///croot/exceptiongroup_1706031385326/work executing @ file:///opt/conda/conda-bld/executing_1646925071911/work fastjsonschema @ file:///opt/conda/conda-bld/python-fastjsonschema_1661371079312/work filelock==3.14.0 flash-attn==2.6.3 fonttools==4.53.0 fpdf==1.7.2 frozenlist==1.4.1 fsspec==2024.5.0 greenlet==3.0.3 huggingface-hub==0.23.3 idna @ file:///croot/idna_1714398848350/work importlib-metadata @ file:///croot/importlib_metadata-suite_1704813515092/work importlib-resources @ file:///croot/importlib_resources-suite_1704281845041/work interegular==0.3.3 ipykernel @ file:///croot/ipykernel_1705933831282/work ipython @ file:///croot/ipython_1691532092695/work ipywidgets @ file:///croot/ipywidgets_1709574692113/work jedi @ file:///tmp/build/80754af9/jedi_1644315233700/work Jinja2 @ file:///croot/jinja2_1716993405101/work joblib==1.4.2 json5 @ file:///tmp/build/80754af9/json5_1624432770122/work json_repair==0.26.0 jsonformer==0.12.0 jsonpatch==1.33 jsonpointer==3.0.0 jsonschema @ file:///croot/jsonschema_1699041609003/work jsonschema-specifications @ file:///croot/jsonschema-specifications_1699032386549/work jupyter @ file:///croot/jupyter_1707947101020/work jupyter-console @ file:///croot/jupyter_console_1679999630278/work jupyter-events @ file:///croot/jupyter_events_1699282461638/work jupyter-lsp @ file:///croot/jupyter-lsp-meta_1699978238815/work jupyter_client @ file:///croot/jupyter_client_1699455897726/work jupyter_core @ file:///croot/jupyter_core_1698937308754/work jupyter_server @ file:///croot/jupyter_server_1699466442171/work jupyter_server_terminals @ file:///croot/jupyter_server_terminals_1686870725608/work jupyterlab @ file:///croot/jupyterlab_1706802623017/work jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work jupyterlab-widgets @ file:///croot/jupyterlab_widgets_1709322880313/work jupyterlab_server @ file:///croot/jupyterlab_server_1699555425460/work kiwisolver==1.4.5 langchain==0.2.5 langchain-core==0.2.9 langchain-text-splitters==0.2.1 langsmith==0.1.81 lark==1.2.2 Levenshtein==0.25.1 llvmlite==0.41.1 lxml==5.2.2 MarkupSafe @ file:///croot/markupsafe_1704205993651/work matplotlib==3.7.5 matplotlib-inline @ file:///opt/conda/conda-bld/matplotlib-inline_1662014470464/work mistune @ file:///opt/conda/conda-bld/mistune_1661496219659/work mpmath==1.3.0 multidict==6.0.5 multiprocess==0.70.16 nbclient @ file:///croot/nbclient_1698934205032/work nbconvert @ file:///croot/nbconvert_1699022732553/work nbformat @ file:///croot/nbformat_1694616755618/work nest-asyncio @ file:///croot/nest-asyncio_1708532673751/work networkx==3.1 nltk==3.8.1 notebook @ file:///croot/notebook_1708029864779/work notebook_shim @ file:///croot/notebook-shim_1699455894279/work numba==0.58.1 numpy==1.24.4 nvidia-cublas-cu12==12.1.3.1 nvidia-cuda-cupti-cu12==12.1.105 nvidia-cuda-nvrtc-cu12==12.1.105 nvidia-cuda-runtime-cu12==12.1.105 nvidia-cudnn-cu12==8.9.2.26 nvidia-cufft-cu12==11.0.2.54 nvidia-curand-cu12==10.3.2.106 nvidia-cusolver-cu12==11.4.5.107 nvidia-cusparse-cu12==12.1.0.106 nvidia-nccl-cu12==2.20.5 nvidia-nvjitlink-cu12==12.5.40 nvidia-nvtx-cu12==12.1.105 openpyxl==3.1.5 orjson==3.10.5 outlines==0.0.46 overrides @ file:///croot/overrides_1699371140756/work packaging @ file:///croot/packaging_1710807400464/work pandas==2.0.3 pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work parso @ file:///opt/conda/conda-bld/parso_1641458642106/work pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work pillow==10.3.0 pkgutil_resolve_name @ file:///croot/pkgutil-resolve-name_1704297459416/work platformdirs @ file:///croot/platformdirs_1692205439124/work ply==3.11 prometheus-client @ file:///tmp/abs_d3zeliano1/croots/recipe/prometheus_client_1659455100375/work prompt-toolkit @ file:///croot/prompt-toolkit_1704404351921/work protobuf==5.27.2 psutil @ file:///opt/conda/conda-bld/psutil_1656431268089/work ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work pyairports==2.1.1 pyarrow==16.1.0 pyarrow-hotfix==0.6 pycocoevalcap==1.2 pycocotools==2.0.7 pycountry==24.6.1 pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work pydantic==2.7.4 pydantic_core==2.18.4 Pygments @ file:///croot/pygments_1684279966437/work pyparsing==3.1.2 PyQt5==5.15.10 PyQt5-sip @ file:///croot/pyqt-split_1698769088074/work/pyqt_sip PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work python-dateutil @ file:///croot/python-dateutil_1716495738603/work python-docx==1.1.2 python-json-logger @ file:///croot/python-json-logger_1683823803357/work python-Levenshtein==0.25.1 pytz @ file:///croot/pytz_1713974312559/work PyYAML @ file:///croot/pyyaml_1698096049011/work pyzmq @ file:///croot/pyzmq_1705605076900/work qtconsole @ file:///croot/qtconsole_1709231153903/work QtPy @ file:///croot/qtpy_1700144840038/work rapidfuzz==3.9.3 referencing @ file:///croot/referencing_1699012038513/work regex==2024.5.15 requests @ file:///croot/requests_1716902831423/work responses==0.18.0 rfc3339-validator @ file:///croot/rfc3339-validator_1683077044675/work rfc3986-validator @ file:///croot/rfc3986-validator_1683058983515/work rouge==1.0.1 rouge_score==0.1.2 rpds-py @ file:///croot/rpds-py_1698945930462/work safetensors==0.4.3 scikit-learn==1.3.2 scipy==1.10.1 seaborn==0.13.2 Send2Trash @ file:///croot/send2trash_1699371139552/work sentence-transformers==3.0.1 sentencepiece==0.2.0 sip @ file:///croot/sip_1698675935381/work six @ file:///tmp/build/80754af9/six_1644875935023/work sniffio @ file:///croot/sniffio_1705431295498/work sns==0.1 soupsieve @ file:///croot/soupsieve_1696347547217/work SQLAlchemy==2.0.31 stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work sympy==1.12.1 tabulate==0.9.0 tenacity==8.4.1 termcolor==2.4.0 terminado @ file:///croot/terminado_1671751832461/work threadpoolctl==3.5.0 tiktoken==0.7.0 timm==1.0.8 tinycss2 @ file:///croot/tinycss2_1668168815555/work tokenizers==0.19.1 tomli @ file:///opt/conda/conda-bld/tomli_1657175507142/work torch==2.3.1 torchvision==0.18.1 tornado @ file:///croot/tornado_1696936946304/work tqdm==4.66.4 traitlets @ file:///croot/traitlets_1671143879854/work transformers==4.40.2 transformers-stream-generator==0.0.5 triton==2.3.1 typing_extensions @ file:///croot/typing_extensions_1715268824938/work tzdata==2024.1 urllib3 @ file:///croot/urllib3_1715635851070/work wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work webencodings==0.5.1 websocket-client @ file:///croot/websocket-client_1715878298792/work widgetsnbextension @ file:///croot/widgetsnbextension_1709322880396/work xxhash==3.4.1 yarl==1.9.4 zipp @ file:///croot/zipp_1704206909481/work ```

Context for the issue:

No response

dottxt-ai / outlines