search

Center-for-Digital-Narrative / elmcip

Electronic Literature as a Model of Creativity and Innovation in Practice (ELMCIP) is a collaborative research project funded by Humanities in the European Research Area (HERA) JRP for Creativity and Innovation built in Drupal
https://elmcip.net
4 stars 0 forks source link

Taxonomy record_status lists never finish #217

Closed steinmb closed 8 years ago

steinmb commented 8 years ago

Flushed out this while going through logs trying to track down #200 and #216. Typical triggered here by search engines spiders hitting these addresses.

Example

 "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

This put a huge load on the systems. CPU peaks for as long PHP is allowed to run. Currently for 30 seconds before it gives up. Spiders will often return after a few min/sec and try again.

91 is about us trying to fix other stuff with our taxonomy terms. We have this problem both on test and in production so it did not get fixed as a foreseen side effect.

Taxonomy vocabulary

record_status (/admin/structure/taxonomy/record_status)

Suggested fix

TBD

How to test

Hit the following url. They should all return data without giving you a WSOD.

    66.249.76.2 - - [28/Aug/2016:06:36:51 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.50 - - [28/Aug/2016:08:14:28 +0200] "GET /category/gender/male HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    77.75.76.171 - - [28/Aug/2016:08:36:32 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
    66.249.76.58 - - [28/Aug/2016:14:11:26 +0200] "GET /category/gender/male HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [28/Aug/2016:14:11:29 +0200] "GET /category/record-status/not-yet-reviewed HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.69.89 - - [28/Aug/2016:19:27:54 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [28/Aug/2016:19:42:55 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.58 - - [28/Aug/2016:20:06:47 +0200] "GET /category/gender/male HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    180.76.15.31 - - [28/Aug/2016:21:36:11 +0200] "GET /category/gender/female?type=All&taxonomy_vocabulary_1_tid=&order=title&sort=asc HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
    77.75.77.11 - - [28/Aug/2016:23:28:03 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
    66.249.76.2 - - [29/Aug/2016:00:51:13 +0200] "GET /taxonomy/term/1490/all?type=All&page=10 HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [29/Aug/2016:01:03:33 +0200] "GET /category/language/english?type=All&page=8 HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    157.55.39.28 - - [29/Aug/2016:01:15:34 +0200] "GET /sites/default/files/styles/large/public/flyer-outlines.jpg?itok=NeCm_P0T HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    5.102.173.71 - - [29/Aug/2016:01:34:32 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)"
    66.249.76.2 - - [29/Aug/2016:02:49:13 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [29/Aug/2016:02:49:19 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.57 - - [29/Aug/2016:02:50:13 +0200] "GET /category/gender/male HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.4 - - [29/Aug/2016:04:26:24 +0200] "GET /category/gender/female HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.59 - - [29/Aug/2016:08:27:16 +0200] "GET /category/gender/male HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [29/Aug/2016:12:23:11 +0200] "GET /category/gender/female HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    83.161.237.117 - - [29/Aug/2016:15:03:14 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "https://elmcip.net/person/daria-tsoupikova" "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
    66.249.76.2 - - [29/Aug/2016:15:12:12 +0200] "GET /category/record-status/not-yet-reviewed HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.4 - - [29/Aug/2016:15:12:11 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.4 - - [29/Aug/2016:15:13:08 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [29/Aug/2016:15:28:40 +0200] "GET /category/gender/male?type=All&taxonomy_vocabulary_1_tid=&page=1 HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    180.76.15.24 - - [29/Aug/2016:17:40:11 +0200] "GET /category/record-status/approved-record HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
    66.249.76.2 - - [29/Aug/2016:17:50:44 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [29/Aug/2016:20:05:47 +0200] "GET /category/gender/female HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    77.75.78.160 - - [29/Aug/2016:21:20:15 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
    66.249.76.2 - - [29/Aug/2016:22:58:52 +0200] "GET /category/record-status/not-yet-reviewed HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [30/Aug/2016:02:05:36 +0200] "GET /category/gender/female HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    207.46.13.83 - - [30/Aug/2016:03:38:38 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    180.76.15.32 - - [30/Aug/2016:04:13:39 +0200] "GET /category/gender/female?type=All&taxonomy_vocabulary_1_tid=&order=title&sort=asc HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
    66.249.76.2 - - [30/Aug/2016:05:17:11 +0200] "GET /category/record-status/not-yet-reviewed HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [30/Aug/2016:05:22:46 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    84.210.221.145 - - [30/Aug/2016:07:56:19 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "https://elmcip.net/creative-work/derivepedia" "Mozilla/5.0 (iPad; CPU OS 9_3_4 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13G35 Safari/601.1"
    84.210.221.145 - - [30/Aug/2016:07:57:01 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "https://elmcip.net/creative-work/derivepedia" "Mozilla/5.0 (iPad; CPU OS 9_3_4 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13G35 Safari/601.1"
    84.210.110.220 - - [30/Aug/2016:11:47:00 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "https://elmcip.net/critical-writing/hyper-and-deep-attention" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0"
    66.249.76.2 - - [31/Aug/2016:02:44:23 +0200] "GET /category/gender/male HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [31/Aug/2016:05:54:10 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [31/Aug/2016:08:47:42 +0200] "GET /category/gender/male HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [31/Aug/2016:14:32:56 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [31/Aug/2016:17:44:19 +0200] "GET /taxonomy/term/1490/all?type=All&page=10 HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    5.102.173.71 - - [31/Aug/2016:19:51:32 +0200] "GET /category/gender/female HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)"
    66.249.76.2 - - [31/Aug/2016:20:12:33 +0200] "GET /category/language/english?type=All&page=8 HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [31/Aug/2016:20:14:00 +0200] "GET /category/gender/male HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.2 - - [31/Aug/2016:20:14:11 +0200] "GET /category/language/english HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.64.216 - - [31/Aug/2016:20:33:44 +0200] "GET /category/language/english?type=All&page=9 HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    68.180.228.219 - - [31/Aug/2016:21:57:04 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
    68.180.228.219 - - [31/Aug/2016:21:57:35 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
    68.180.228.219 - - [31/Aug/2016:21:58:06 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
    68.180.228.219 - - [31/Aug/2016:21:58:37 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
    68.180.228.219 - - [31/Aug/2016:21:59:39 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
    31.221.39.139 - - [31/Aug/2016:22:14:35 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0"
    31.221.39.139 - - [31/Aug/2016:22:15:56 +0200] "GET /category/record-status/incomplete-record-stub HTTP/1.1" 500 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0"
steinmb commented 8 years ago

We should in production block these addresses in apache until we know how to serve them up. Testing mod_rewrite locally.

steinmb commented 8 years ago

Closing this as a duplicate. Addressing this in issue #163