algolia / algoliasearch-client-python

⚡️ A fully-featured and blazing-fast Python API client to interact with Algolia.
https://www.algolia.com/doc/api-client/getting-started/install/python/?language=python
MIT License
196 stars 67 forks source link

feat: Edit the requests configuration pool_maxsize variable when initializing client #521

Open lautarovc opened 3 years ago

lautarovc commented 3 years ago

Description

I'm replicating a dataset (60k records) with 100 threads (Python's builtin ThreadExecutor) at a time. And I find that from time to time I keep getting these warnings from the 'requests' library: "Connection pool is full, discarding connection: xxxxxxxx.algolia.net".

The ability to edit the _poolmaxsize configuration variable when initializing the search client would solve this kind of issues, perhaps even allowing to edit other requests configurations.

Steps To Reproduce

Initialize a ThreadExecutor of 100 max workers. Create a function that uses 'save_object' to store a record in a predefined index. Loop through thousands of records, for each record: -Add the function that saves the record with the record data to the executor

You should get the warning every ~1000 records.

rgs258 commented 1 year ago

I'm seeing the same thing. As a workaround, I'm initializing the client in the callable. Would love to see this taken up!