search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.17k stars 751 forks source link

replace requests with httpx #213

Closed singingwolfboy closed 7 months ago

singingwolfboy commented 8 months ago

HTTPX is a HTTP client for Python 3, that seamlessly supports both synchronous and asynchronous workflows. The API is broadly compatible with requests, but it's entirely rebuilt; it shares no dependencies at all with requests.

Tiktoken requires Python 3.8 or higher, so the fact that HTTPX only supports Python 3 is not a problem. It also means that the developers who use tiktoken may want to take advantage of syntax and features that are only present in Python 3, like async support. Currently, tiktoken pulls in requests as a dependency even if the developer would prefer to use httpx for their application code -- which is not terrible, but more dependencies means more to download, slower CI execution, and a potentially larger attack surface from a security perspective.

Of course, the opposite would also be true for this pull request: if tiktoken pulls in httpx as a dependency, and the developer would prefer to use requests for their application code, we have the same problem. We could specify both requests and httpx as optional dependencies, and use whichever one we manage to import successfully; but that would make the code more complex, and could confuse developers who don't realize that they must install one or the other of these optional dependencies in order to use tiktoken. As a result, I decided to start with a pull request suggesting the simplest possible change: directly swapping requests for httpx.

Please let me know what you think of this idea! I'm happy to alter this pull request in response to feedback.

hauntsaninja commented 7 months ago

Thanks, but as you say, the situation is symmetric, so there's no reason to use httpx over requests. I don't think optional dependencies for core functionality is worth the cost. I'd recommend forking if you need this level of control over your dependencies.