Hello, I have a memory issue when using replace_all_objects. When using this function with a significant amount of documents (5 Million), I use an iterator to minimize memory consumption.
I expect the memory usage to stay flat during the operation, however it keeps increasing. (cf image below)
Hello, I have a memory issue when using
replace_all_objects
. When using this function with a significant amount of documents (5 Million), I use an iterator to minimize memory consumption. I expect the memory usage to stay flat during the operation, however it keeps increasing. (cf image below)Upon investigation, it looks like the cause of this memory usage increase comes from the function
SearchIndex._chunk
, and more specifically the listraw_responses
, which stores responses for every request sent. https://github.com/algolia/algoliasearch-client-python/blob/3bb9108d9dff627f12c921ad23dab02984f70a44/algoliasearch/search_index.py#L505-L528This is a problem because the response of
/1/indexes/{indexName}/batch
contains the list ofobjectIDs
With 5M documents, each with an objectID of ~15 characters, this accounts for 300MB.
Is there a request_option for the API not to return
objectIDs
, or for the code not to store them inraw_responses
?Thank you 🙏