pallets-eco / cachelib

Extract from werkzeug.cache
BSD 3-Clause "New" or "Revised" License
134 stars 44 forks source link

Cachelib on heavy load #360

Open Liphtier opened 4 months ago

Liphtier commented 4 months ago

Flask-Session uses Cachelib for filesystem session storage

A Flask based webapp was tested with Apache Benchmark tool, sending 1000 requests with concurrency 100. After approx. 500 requests, apache logs start being flooded with warnings like this:

[Wed Mar 20 19:03:07.814067 2024] [wsgi:error] [pid 23776] [client 10.0.11.184:45501] Traceback (most recent call last):
[Wed Mar 20 19:03:07.814068 2024] [wsgi:error] [pid 23776] [client 10.0.11.184:45501]   File "/usr/local/lib/python3.8/site-packages/cachelib/file.py", line 122, in _remove_expired
[Wed Mar 20 19:03:07.814069 2024] [wsgi:error] [pid 23776] [client 10.0.11.184:45501]     expires = struct.unpack("I", f.read(4))[0]
[Wed Mar 20 19:03:07.814071 2024] [wsgi:error] [pid 23776] [client 10.0.11.184:45501] struct.error: unpack requires a buffer of 4 bytes

To reproduce use Flask-Session in a web app then test it with

ab -n 1000 -c 100 http://web-app.addr

and check error logs

Environment:

Not sure, but may be ulimit for open files is affecting

su - apache -s /bin/bash  -c 'ulimit -Ha' | grep 'open files' 
open files                      (-n) 4096

su - apache -s /bin/bash  -c 'ulimit -Sa' | grep 'open files' 
open files                      (-n) 1024