search

ArgLab / ArgLab_writing_observer

Writing Observer and Learning Observer: A system for monitoring learning process data, with an initial focus on writing process data from Google Docs.
GNU Affero General Public License v3.0
3 stars 2 forks source link

Project: Redis Connection Pool under AsyncIO #117

Open DrLynch opened 1 month ago

DrLynch commented 1 month ago

Based upon recent system tests it appears that we are spending a lot of time on redis connection requests under the async calls. Some of that has to do with the ping and may need to be moved around but it is also worth checking whether the redis-asyncio system is also using pools and is able to handle pool connections. At present we do not explicitly use or manage the connection pooling nor do we turn on more than one connection. We need to check if that is doable and then to check what changes can be made.

DrLynch commented 3 weeks ago

Check on default pool size/behavior and whether changing pool affects connections.

DrLynch commented 3 weeks ago

From past work it seems this is happening by default. What we don't know yet is whether this makes a major difference in operations. That should be tested with experimentation.

Brendon-Hablutzel commented 3 weeks ago

The Redis client we use, redis-py, internally initializes a connection pool with 2^31 maximum connections by default (see source code for reference here).