search

commonsense / conceptnet5

Code for building ConceptNet from raw data.
Other
2.78k stars 355 forks source link

Rate limiting under high load cause error #308

Open amirouche opened 3 years ago

amirouche commented 3 years ago

Under high load, flask-limiter inside limits will raise an exception because no more thread are available:

Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]: [2021-08-16 14:18:39,630] ERROR in app: Exception on /c/zh/文件夾 [GET]
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]: Traceback (most recent call last):
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     response = self.full_dispatch_request()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     rv = self.handle_user_exception(e)
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     return cors_after_request(app.make_response(f(*args, **kwargs)))
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     reraise(exc_type, exc_value, tb)
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     raise value
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1948, in full_dispatch_request
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     rv = self.preprocess_request()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 2242, in preprocess_request
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     rv = func()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask_limiter/extension.py", line 534, in __check_request_limit
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     six.reraise(*sys.exc_info())
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/six.py", line 703, in reraise
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     raise value
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask_limiter/extension.py", line 517, in __check_request_limit
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     self.__evaluate_limits(endpoint, all_limits)
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask_limiter/extension.py", line 393, in __evaluate_limits
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     if not self.limiter.hit(lim.limit, *args):
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/limits/strategies.py", line 132, in hit
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     self.storage().incr(item.key_for(*identifiers), item.get_expiry())
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/limits/storage.py", line 171, in incr
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     self.__schedule_expiry()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/limits/storage.py", line 159, in __schedule_expiry
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     self.timer.start()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/usr/lib/python3.6/threading.py", line 846, in start
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     _start_new_thread(self._bootstrap, ())
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]: RuntimeError: can't start new thread
amirouche commented 3 years ago

ref: https://github.com/commonsense/conceptnet5/issues/306 ref: https://limits.readthedocs.io/en/stable/ ref: https://github.com/alisaifee/flask-limiter/issues

amirouche commented 3 years ago

The approach taken by flask-limiter via limits is wrong in this context. Spawning a POSIX thread for every unique client by IP is doomed to fail. One way to workaround it is to investigate REDIS backend in flask-limiter / limits OR rely on nginx [0] or apache rate limiting.

[0] https://www.nginx.com/blog/rate-limiting-nginx/

amirouche commented 3 years ago

It is preferable to use apache or nginx rate limiting feature.