search

kobotoolbox / kobocat

Our (backend) server for providing blank forms to Collect and Enketo and for receiving and storing submissions.
BSD 2-Clause "Simplified" License
117 stars 127 forks source link

Throttle submissions #626

Open jnm opened 4 years ago

jnm commented 4 years ago

Having single users try to submit 250,000 submissions per day seems to overwhelm Postgres running on an AWS EC2 r5.2xlarge. We need a way to reject submissions that occur too frequently. This will probably have to be done by form owner, not by authenticated user or client IP.

jnm commented 4 years ago

I think I'll use https://github.com/soulomoon/python-throttle Also of interest: https://github.com/sukobuto/python-redis-gt

All of this stuff only supports Python 3, but it shouldn't be hard to backport (edit: ha, ha, right). Another option if my port falls on its face is https://web.archive.org/web/20190128010148/http://flask.pocoo.org/snippets/70/.

There are plug-and-play DRF packages, but submissions isn't a DRF endpoint. There are some more generic Django throttling packages, but I haven't found one that I like.

jnm commented 4 years ago

Using https://github.com/jnm/python-throttle under Python 2 Deployed https://github.com/kobotoolbox/kobocat/tree/two-databases_2.019.47__hotfix__throttle to OCHA

jnm commented 4 years ago

Continued progress (based on the outdated branch that OCHA still uses): https://github.com/kobotoolbox/kobocat/compare/two-databases_2.019.47__hotfix...two-databases_2.019.47__hotfix__throttle