search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.52k stars 856 forks source link

Remove unused imports #348

Closed esadek closed 1 month ago

esadek commented 1 month ago

Remove unused imports from scripts/benchmark.py and tests/test_helpers.py.

hauntsaninja commented 1 month ago

Thanks, but we use the imports in the internal version of tiktoken (the relevant code is redacted for open source). The scripts folder isn't packaged, so these don't affect users

esadek commented 1 month ago

Thanks for the quick reply @hauntsaninja. I'm curious, if the relevant code is redacted for open source wouldn't it make sense to redact these import as well?

hauntsaninja commented 1 month ago

Maybe? I don't think these imports leak very much information. If there was infinite time, it'd make sense to set up the redaction, but there isn't — I don't get much time to work on tiktoken as it is.