search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.06k stars 749 forks source link

tiktoken 0.7.0 isn't compatible with python 3.11.* #300

Closed trenton3983 closed 1 month ago

trenton3983 commented 1 month ago

python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;

└─ tiktoken 0.7.0**  is not installable because there are no viable options
   ├─ tiktoken 0.7.0 conflicts with any installable versions previously reported;
   ├─ tiktoken 0.7.0 would require
   │  └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
   ├─ tiktoken 0.7.0 would require
   │  └─ python >=3.12,<3.13.0a0 , which conflicts with any installable versions previously reported;
   ├─ tiktoken 0.7.0 would require
   │  └─ python >=3.8,<3.9.0a0 , which conflicts with any installable versions previously reported;
   └─ tiktoken 0.7.0 would require
      └─ python >=3.9,<3.10.0a0 , which conflicts with any installable versions previously reported.

This is a conflict tree from conda in an environment with pyhton 3.11.9

kalgoritmi commented 1 month ago

What channels does your conda configuration use? If you add -c 'conda-forge' to your conda install command, it should install tiktoken without a problem.

This is not a tiktoken incompatibility issue, it is related to the unavailability of a built tiktoken package for the conda defaults channel, this is a different channel from the community driven 'conda-forge'.

If you want to test it with a new environment, run this:

conda create -n conda_forge_channel -c 'conda-forge' python=3.11.9 tiktoken

whereas, this fails:

conda create -n default_channel python=3.11.9 tiktoken

To permanently add a channel for use with all environments, use this:

conda config --add channels conda-forge

Besides these, you can see that tiktoken works with python 3.11.*, if you just create an environment, and then pip install it:

conda create -n python_3.11.9 python=3.11.9
conda deactivate && conda activate python_3.11.9
pip install tiktoken==0.7.0

This is because there is an already built wheel for CPython311 and tiktoken0.7.0 on PyPI simple for most architectures see this link, even if there was no available pre-built wheel, it would have been built from sources when running pip install.

trenton3983 commented 1 month ago

This is not a tiktoken incompatibility issue, it is related to the unavailability of a built tiktoken package for the conda defaults channel.

This is not the issue.

Based on the conflict tree, it should be clear this is not an issue with the channel, because tiktoken 0.7.0 would not be present for the defaults conda channels.

Specifying a version not available in the channel would result in:

PackagesNotFoundError: The following packages are not available from current channels:

  - tiktoken=0.7.0*

Current channels:

  - defaults

In any case, my first channel in .condarc is conda-forge, which I verified.

channels:
  - conda-forge
  - defaults
hauntsaninja commented 1 month ago

I don't know very much about conda / I don't provide conda builds. tiktoken supports 3.11 and you can find 3.11 wheels on PyPI: https://pypi.org/project/tiktoken/0.7.0/#files