search

ho-nl / vagrant-development-box

Maintainer: Paul
MIT License
0 stars 0 forks source link

Remove rate limit from box #73

Open NickdeK opened 6 years ago

NickdeK commented 6 years ago

Can sometimes result in a "too many requests" error. Should change /data/web/nginx/http.conn_ratelimit to the following content (courtesy of @JeroenVanLeusden):

map $remote_addr $conn_limit_map {
  default '';
}

Might also be interesting to remove or disable the bots zone for in the nginx config to enable local crawling tests (but that might be a to occasional scenario to implement).

NickdeK commented 5 years ago

This still seems to be a issue. When using PWA Studio the rate limiting kicks in, in this case the rate limiting setup for bots.

Workaround: edit/add /data/web/nginx/http.ratelimit with the following contents:

map $http_user_agent $limit_bots {
default '';
~*(node-fetch|google|bing|heartbeat|uptimerobot|shoppimon|facebookexternal|monitis.com|Zend_Http_Client|magereport.com|SendCloud/|Adyen|contentkingapp) '';
~*(http|crawler|spider|bot|search|Wget/|Python-urllib|PHPCrawl|bGenius|MauiBot) 'bot';
}

Adding node-fetch to the first list prevents the blocking of the api requests from PWA Studio.

Also see: https://support.hypernode.com/knowledgebase/resolving-429-many-requests/#Rate_limiting_against_bots_and_crawlers