relax tokenizers requirement

[efriis]

This PR updates the poetry.lock and pyproject.toml files.

poetry.lock

The `poetry.lock file has been updated to use Poetry version 1.7.1. The following packages have also been updated:

• boto3 from version 1.34.121 to 1.34.126
• botocore from version 1.34.121 to 1.34.126
• filelock from version 3.14.0 to 3.15.1
• huggingface-hub from version 0.23.3 to 0.23.4
• packaging from version 24.0 to 24.1
• pydantic from version 2.7.3 to 2.7.4
• tokenizers from version 0.15.2 to 0.19.1
• typing-extensions from version 4.12.1 to 4.12.2

pyproject.toml

The pyproject.toml file has been updated to use Poetry version 5.5.8. Thetokenizers` package has also been updated to use version 0.15 or above, but below version 1.

[CLAassistant]

All committers have signed the CLA.

[billytrend-cohere]

Thanks, this is resolved in the latest release