mistralai / mistral-common

Apache License 2.0
651 stars 69 forks source link

[BUG: Tokenizer Decode error during function calling] #44

Open Mohrizk opened 2 months ago

Mohrizk commented 2 months ago

Python -VV

Python 3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) [GCC 12.3.0]

Pip Freeze

absl-py @ file:///home/conda/feedstock_root/build_artifacts/absl-py_1705494584803/work
aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1713964843148/work
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
annotated-types @ file:///home/conda/feedstock_root/build_artifacts/annotated-types_1716290248287/work
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1717693030552/work
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1692818318753/work
argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1695386553988/work
arrow @ file:///home/conda/feedstock_root/build_artifacts/arrow_1696128962909/work
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1698341106958/work
astunparse @ file:///home/conda/feedstock_root/build_artifacts/astunparse_1610696312422/work
async-lru @ file:///home/conda/feedstock_root/build_artifacts/async-lru_1690563019058/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1704011227531/work
Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1702422572539/work
beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1705564648255/work
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1696630167146/work
blinker @ file:///home/conda/feedstock_root/build_artifacts/blinker_1715091184126/work
blis @ file:///home/conda/feedstock_root/build_artifacts/cython-blis_1696148823813/work
Bottleneck @ file:///work/ci_py311/bottleneck_1676827011612/work
Brotli @ file:///home/conda/feedstock_root/build_artifacts/brotli-split_1695989787169/work
cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1708987703938/work
catalogue @ file:///home/conda/feedstock_root/build_artifacts/catalogue_1695626386490/work
certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1718025014955/work/certifi
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1696001724357/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1698833585322/work
click @ file:///home/conda/feedstock_root/build_artifacts/click_1692311806742/work
cloudpathlib @ file:///home/conda/feedstock_root/build_artifacts/cloudpathlib-meta_1708995864616/work
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1666700638685/work
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1710320294760/work
confection @ file:///home/conda/feedstock_root/build_artifacts/confection_1701179076994/work
contourpy @ file:///home/conda/feedstock_root/build_artifacts/contourpy_1712429897138/work
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography-split_1717559408348/work
cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1696677705766/work
cymem @ file:///home/conda/feedstock_root/build_artifacts/cymem_1695443490958/work
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1719378645730/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
docstring_parser==0.16
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1704921103267/work
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1698579936712/work
fastai @ file:///home/jhoward/miniconda3/conda-bld/fastai_1714256075359/work
fastcore @ file:///Users/jhoward/miniconda3/conda-bld/fastcore_1719125228499/work
fastdownload @ file:///home/jhoward/mambaforge/conda-bld/fastdownload_1657219113869/work
fastjsonschema @ file:///home/conda/feedstock_root/build_artifacts/python-fastjsonschema_1718477020893/work/dist
fastprogress @ file:///home/jhoward/mambaforge/conda-bld/fastprogress_1658473398631/work
filelock @ file:///home/conda/feedstock_root/build_artifacts/filelock_1719088281970/work
fire==0.6.0
flatbuffers @ file:///home/conda/feedstock_root/build_artifacts/python-flatbuffers_1711466727397/work
fonttools @ file:///home/conda/feedstock_root/build_artifacts/fonttools_1720359030008/work
fqdn @ file:///home/conda/feedstock_root/build_artifacts/fqdn_1638810296540/work/dist
frozenlist @ file:///home/conda/feedstock_root/build_artifacts/frozenlist_1702645450877/work
fsspec==2024.6.1
gast @ file:///home/conda/feedstock_root/build_artifacts/gast_1719403123000/work
gmpy2 @ file:///home/conda/feedstock_root/build_artifacts/gmpy2_1715527288181/work
google-auth @ file:///home/conda/feedstock_root/build_artifacts/google-auth_1719906113316/work
google-auth-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/google-auth-oauthlib_1688235217226/work
google-pasta==0.2.0
greenlet==3.0.3
grpcio @ file:///home/conda/feedstock_root/build_artifacts/grpc-split_1690942284331/work
h11 @ file:///home/conda/feedstock_root/build_artifacts/h11_1664132893548/work
h2 @ file:///home/conda/feedstock_root/build_artifacts/h2_1634280454336/work
h5py @ file:///home/conda/feedstock_root/build_artifacts/h5py_1717664837666/work
hpack==4.0.0
httpcore @ file:///home/conda/feedstock_root/build_artifacts/httpcore_1711596990900/work
httpx @ file:///home/conda/feedstock_root/build_artifacts/httpx_1708530890843/work
huggingface-hub==0.24.6
hyperframe @ file:///home/conda/feedstock_root/build_artifacts/hyperframe_1619110129307/work
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1713279365350/work
import-ipynb==0.1.4
importlib_metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1719361860083/work
importlib_resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1711040877059/work
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1719845459717/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1719582526268/work
ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1716897651763/work
isoduration @ file:///home/conda/feedstock_root/build_artifacts/isoduration_1638811571363/work/dist
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1715127149914/work
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1714665484399/work
json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1712986206667/work
jsonpatch==1.33
jsonpointer @ file:///home/conda/feedstock_root/build_artifacts/jsonpointer_1718283368205/work
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1714573116818/work
jsonschema-specifications @ file:///tmp/tmpkv1z7p57/src
jupyter @ file:///home/conda/feedstock_root/build_artifacts/jupyter_1696255489086/work
jupyter-console @ file:///home/conda/feedstock_root/build_artifacts/jupyter_console_1678118109161/work
jupyter-events @ file:///home/conda/feedstock_root/build_artifacts/jupyter_events_1710805637316/work
jupyter-lsp @ file:///home/conda/feedstock_root/build_artifacts/jupyter-lsp-meta_1712707420468/work/jupyter-lsp
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1716472197302/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1710257359434/work
jupyter_server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1717122053158/work
jupyter_server_terminals @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_terminals_1710262634903/work
jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1719418677442/work
jupyterlab_pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1707149102966/work
jupyterlab_server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server-split_1716433953404/work
jupyterlab_widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1716891641122/work
keras @ file:///home/conda/feedstock_root/build_artifacts/keras_1698427100715/work/keras-2.14.0-py3-none-any.whl#sha256=d7429d1d2131cc7eb1f2ea2ec330227c7d9d38dab3dfdf2e78defee4ecc43fcd
kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1695379920604/work
langchain==0.2.16
langchain-core==0.2.38
langchain-text-splitters==0.2.4
langcodes @ file:///home/conda/feedstock_root/build_artifacts/langcodes_1714235526219/work
langsmith==0.1.114
language_data @ file:///home/conda/feedstock_root/build_artifacts/language-data_1714193818885/work
marisa-trie @ file:///home/conda/feedstock_root/build_artifacts/marisa-trie_1706566451123/work
Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1710435156458/work
markdown-it-py @ file:///home/conda/feedstock_root/build_artifacts/markdown-it-py_1686175045316/work
MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1706899926732/work
matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1715976244352/work
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1713250518406/work
mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1704317613764/work
mistral_common==1.3.4
mistral_inference==1.3.1
mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1698947099619/work
ml-dtypes @ file:///home/conda/feedstock_root/build_artifacts/ml_dtypes_1695280940315/work
mpmath @ file:///home/conda/feedstock_root/build_artifacts/mpmath_1678228039184/work
multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1707040702345/work
munkres==1.1.4
murmurhash @ file:///home/conda/feedstock_root/build_artifacts/murmurhash_1695449783148/work
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1710317608672/work
nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert-meta_1718135430380/work
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1712238998817/work
nbimporter==0.3.4
nest_asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1705850609492/work
networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1712540363324/work
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1717767745914/work
notebook_shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1707957777232/work
numexpr @ file:///home/conda/feedstock_root/build_artifacts/numexpr_1716812153765/work
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1707225376651/work/dist/numpy-1.26.4-cp311-cp311-linux_x86_64.whl#sha256=d08e1c9e5833ae7780563812aa73e2497db1ee3bd5510d3becb8aa18aa2d0c7c
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.6.68
nvidia-nvtx-cu12==12.1.105
oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1666056362788/work
opt-einsum @ file:///home/conda/feedstock_root/build_artifacts/opt_einsum_1696448916724/work
orjson==3.10.7
overrides @ file:///home/conda/feedstock_root/build_artifacts/overrides_1706394519472/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1718189413536/work
pandas @ file:///croot/pandas_1697477158611/work/dist/pandas-2.1.1-cp311-cp311-linux_x86_64.whl#sha256=ed5518777956b76c6290791e98ec9ff6ea64a621c757052a7697aa4252bc0d9b
pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1712320355065/work
pathlib==1.0.1
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1706113125309/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1712154447422/work
pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1694617248815/work
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1715777629804/work
ply @ file:///home/conda/feedstock_root/build_artifacts/ply_1712242996588/work
preshed @ file:///home/conda/feedstock_root/build_artifacts/preshed_1695644751322/work
prometheus_client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1707932675456/work
prompt_toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1718047967974/work
protobuf==4.21.12
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1719274586160/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
pyasn1 @ file:///home/conda/feedstock_root/build_artifacts/pyasn1_1713209357222/work
pyasn1_modules @ file:///home/conda/feedstock_root/build_artifacts/pyasn1-modules_1713209683338/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1711811537435/work
pydantic @ file:///home/conda/feedstock_root/build_artifacts/pydantic_1720293063581/work
pydantic_core @ file:///home/conda/feedstock_root/build_artifacts/pydantic-core_1720041200430/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1714846767233/work
PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1706895065046/work
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1706660063483/work
pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1709721012883/work
PyQt5==5.15.9
PyQt5-sip==12.12.2
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1709299778482/work
python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1677079630776/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1706886791323/work
pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work
PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1695373611984/work
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1715024370414/work
qtconsole @ file:///home/conda/feedstock_root/build_artifacts/qtconsole-base_1714942934316/work
QtPy @ file:///home/conda/feedstock_root/build_artifacts/qtpy_1698112029416/work
referencing @ file:///home/conda/feedstock_root/build_artifacts/referencing_1714619483868/work
regex==2024.7.24
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1717057054362/work
requests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1711290127547/work
rfc3339-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3339-validator_1638811747357/work
rfc3986-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3986-validator_1598024191506/work
rich @ file:///home/conda/feedstock_root/build_artifacts/rich-split_1709150387247/work/dist
rpds-py @ file:///home/conda/feedstock_root/build_artifacts/rpds-py_1715090025935/work
rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1658328885051/work
safetensors==0.4.4
scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1719998107382/work/dist/scikit_learn-1.5.1-cp311-cp311-linux_x86_64.whl#sha256=149fab54f11e79470f037b956bb6061954107c264039bb95b7412ab36f06c972
scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy-split_1720323007424/work/dist/scipy-1.14.0-cp311-cp311-linux_x86_64.whl#sha256=1555805d3d22eadcd79d8bbf4de2865c7ad881feceb57d3c2d91ec2469d4acf7
Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1712584999685/work
sentencepiece==0.2.0
shellingham @ file:///home/conda/feedstock_root/build_artifacts/shellingham_1698144360966/work
simple_parsing==0.1.5
sip @ file:///home/conda/feedstock_root/build_artifacts/sip_1697300428897/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
smart_open @ file:///home/conda/feedstock_root/build_artifacts/smart_open_split_1711455333245/work/dist
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1708952932303/work
soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1693929250441/work
spacy @ file:///home/conda/feedstock_root/build_artifacts/spacy_1720075566465/work
spacy-legacy @ file:///home/conda/feedstock_root/build_artifacts/spacy-legacy_1674550301837/work
spacy-loggers @ file:///home/conda/feedstock_root/build_artifacts/spacy-loggers_1694527114282/work
SQLAlchemy==2.0.34
srsly @ file:///home/conda/feedstock_root/build_artifacts/srsly_1695654015181/work
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
sympy @ file:///home/conda/feedstock_root/build_artifacts/sympy_1718625539893/work
tenacity==8.5.0
tensorboard @ file:///home/conda/feedstock_root/build_artifacts/tensorboard_1695917943728/work/tensorboard-2.14.1-py3-none-any.whl#sha256=3db108fb58f023b6439880e177743c5f1e703e9eeb5fb7d597871f949f85fd58
tensorboard-data-server @ file:///home/conda/feedstock_root/build_artifacts/tensorboard-data-server_1695425375375/work/tensorboard_data_server-0.7.0-py3-none-manylinux2014_x86_64.whl#sha256=4a87e32f17958007f01c1acb90cf7aab5877e41b1a929e3a016020697c37b53d
tensorflow @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-split_1699606989644/work/tensorflow_pkg/tensorflow-2.14.0-cp311-cp311-linux_x86_64.whl#sha256=581f1dc8aef4e7318d61e11cac8cefe9ab4066f8caedf02cbeadc93c00c569b9
tensorflow-estimator @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-split_1699606989644/work/tensorflow-estimator/wheel_dir/tensorflow_estimator-2.14.0-py2.py3-none-any.whl#sha256=1aa91c08a19ee48a3fa84b8482235363e5f3dd5741d3373554049a40751b5735
termcolor @ file:///home/conda/feedstock_root/build_artifacts/termcolor_1704357939450/work
terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1710262609923/work
thinc @ file:///home/conda/feedstock_root/build_artifacts/thinc_1715461672215/work
threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1714400101435/work
tiktoken==0.7.0
tinycss2 @ file:///home/conda/feedstock_root/build_artifacts/tinycss2_1713974937325/work
tokenizers==0.19.1
toml @ file:///home/conda/feedstock_root/build_artifacts/toml_1604308577558/work
tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work
torch==2.4.0
torchaudio==2.3.1
torchvision==0.18.1
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1717722848697/work
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1714854870413/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1713535121073/work
transformers==4.44.2
triton==3.0.0
typer==0.12.3
typer-slim==0.12.3
types-python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/types-python-dateutil_1710589910274/work
typing-utils @ file:///home/conda/feedstock_root/build_artifacts/typing_utils_1622899189314/work
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1717802530399/work
tzdata @ file:///croot/python-tzdata_1690578112552/work
uri-template @ file:///home/conda/feedstock_root/build_artifacts/uri-template_1688655812972/work/dist
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1719391292974/work
wasabi @ file:///home/conda/feedstock_root/build_artifacts/wasabi_1715409627083/work
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1704731205417/work
weasel @ file:///home/conda/feedstock_root/build_artifacts/weasel_1719987001521/work
webcolors @ file:///home/conda/feedstock_root/build_artifacts/webcolors_1717667289718/work
webencodings @ file:///home/conda/feedstock_root/build_artifacts/webencodings_1694681268211/work
websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1713923384721/work
Werkzeug @ file:///home/conda/feedstock_root/build_artifacts/werkzeug_1715000201436/work
widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1716891659446/work
wrapt @ file:///home/conda/feedstock_root/build_artifacts/wrapt_1666806105113/work
xformers==0.0.27.post2
yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1705508295175/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1718013267051/work
zstandard==0.22.0

Reproduction Steps

I am working on function calling methods, however i face challenged in the decode function as the output has not been consistent sometimes it produces the parameters required and other it errors as specialtoken policy error despite of turning it to ignore "1"

Here is my snippet, while use Nemo-Instruct-2407:


 model = Transformer.from_folder(setup.mistral_models_path)
 #tokenizer = MistralTokenizer.from_file(f"{setup.mistral_models_path}/tekken.json")
tokenizer = MistralTokenizer.v3(is_tekken=True)
tokenizer.special_token_policy = 1;
get_flow_definition = {
        "type": "function",
        "function": {
            "name": "get_flow",
            "description": "Get a flow using the id",
            "parameters": {
                   "type": "object",
                    "properties": {
                        "flow_id": {
                            "type": "string",
                            "description": "id of the flow",
                        },
                    },
                    "required": ["flow_id"],
                },
        },
    }
trial = [get_flow_definition]
readyTools = []
for tool in trial:
    if isinstance(tool, dict) and 'function' in tool:
        newTool =Tool(
            function = Function(
                name=tool["function"]["name"], 
                description=tool["function"]["description"],
                parameters=tool["function"]["parameters"]
            )
        )
        readyTools.append(newTool)

print(readyTools)

prompt = "can you get me a flow?"

messges = [
    UserMessage(content=prompt)
]
completion_request = ChatCompletionRequest(
    tools=readyTools,
    messages=[UserMessage(content=prompt)],
)

tokens = tokenizer.encode_chat_completion(completion_request).tokens
out_tokens, _ = generate([tokens], model, max_tokens=1024, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
result = tokenizer.decode(out_tokens[0])
print(result)

My issue due to output creates fluctuation, most of the time produces the below error:

`---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 48
     46 out_tokens, _ = generate([tokens], model, max_tokens=1024, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
     47 tokenizer.special_token_policy = 1;
---> 48 result = tokenizer.decode(out_tokens[0])
     49 print(result)

File [~/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/mistral.py:148](http://147.185.40.32:20045/lab/tree/ai-worker/src/v1/ai/model/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/mistral.py#line=147), in MistralTokenizer.decode(self, tokens)
    147 def decode(self, tokens: List[int]) -> str:
--> 148     return self.instruct_tokenizer.decode(tokens)

File [~/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/sentencepiece.py:200](http://147.185.40.32:20045/lab/tree/ai-worker/src/v1/ai/model/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/sentencepiece.py#line=199), in InstructTokenizerBase.decode(self, tokens)
    199 def decode(self, tokens: List[int]) -> str:
--> 200     return self.tokenizer.decode(tokens)

File [~/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/tekken.py:234](http://147.185.40.32:20045/lab/tree/ai-worker/src/v1/ai/model/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/tekken.py#line=233), in Tekkenizer.decode(self, tokens)
    233 def decode(self, tokens: List[int]) -> str:
--> 234     return "".join(self._decode_all(tokens, special_token_policy=self._special_token_policy))

File [~/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/tekken.py:203](http://147.185.40.32:20045/lab/tree/ai-worker/src/v1/ai/model/mambaforge/envs/env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/tekken.py#line=202), in Tekkenizer._decode_all(self, tokens, special_token_policy)
    201 if is_special:
    202     if special_token_policy == SpecialTokenPolicy.RAISE:
--> 203         raise ValueError(
    204             f"Decoding `tokens` that contain special tokens ({list(group)}) is not allowed. \n"
    205             "Either make sure `tokens` do not include any special tokens or, "
    206             "if you want to decode `tokens` that includes special tokens, "
    207             "change the tokenizer's special token policy to IGNORE or KEEP: \n"
    208             "```\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer"
    209             "\nfrom mistral_common.tokens.tokenizers.tekken import SpecialTokenPolicy"
    210             "\n\ntokenizer = MistralTokenizer.v3(is_tekken=True)"
    211             "\ntokenizer.special_token_policy = SpecialTokenPolicy.IGNORE  # or SpecialTokenPolicy.KEEP"
    212             "\n```"
    213         )
    214     elif special_token_policy == SpecialTokenPolicy.KEEP:
    215         decoded.extend(self._all_special_tokens[t] for t in group)

ValueError: Decoding `tokens` that contain special tokens ([9]) is not allowed. 
Either make sure `tokens` do not include any special tokens or, if you want to decode `tokens` that includes special tokens, change the tokenizer's special token policy to IGNORE or KEEP: 

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.tokens.tokenizers.tekken import SpecialTokenPolicy

tokenizer = MistralTokenizer.v3(is_tekken=True) tokenizer.special_token_policy = SpecialTokenPolicy.IGNORE # or SpecialTokenPolicy.KEEP

Expected Behavior

Expected is to produce the params of the flow id required

Additional Context

No response

Suggested Solutions

No response

Mohrizk commented 2 months ago

It consistently worked when i manually removed the special token causes this issue [9] with the below line filtered_tokens = [token for token in out_tokens[0] if token != 9]

I still think we need a valid solution for this, if possible

patrickvonplaten commented 2 months ago

Hey @Mohrizk,

Thanks for the issue - can you please make sure to post a fully, reproducible code snippet that I can copy-paste into a Python shell and it will run correctly.

For the above code snippet, I don't know exactly where you took the model weights from. Also Transformer and MistralTokenizer are not imported so the code snippet is not runable.

Can you try to post a complete code snippet please?