search

mozilla / page-metadata-service

DEPRECATED - A RESTful service that returns the metadata about a given URL.
Mozilla Public License 2.0
19 stars 8 forks source link

Rate limit connections by domain #102

Closed jaredlockhart closed 7 years ago

jaredlockhart commented 7 years ago

We need to track the rate at which we connect to remote hosts/domains and then limit them. Ie we should not be attempting to hit some example.com more than X times per second. If rate limit is exceeded we should pause that work until we are able to complete it. We can use redis to track this effectively.

jaredlockhart commented 7 years ago

This may be better suited to live in teh proxy which already has a redis instance.

jaredlockhart commented 7 years ago

Log aggressive source IPs to facilitate building a malicious caller database.

jaredlockhart commented 7 years ago

This will be handled by

https://github.com/mozilla/embedly-proxy/issues/71