search

Dascienz / phpBB-forum-scraper

Python-based web crawlers for scraping phpBB forum posts.
38 stars 18 forks source link

[NEED HELP] Scrapes 0 items, just crawls, output file 0 byte, logs included #14

Open NoobBugHunter opened 3 years ago

NoobBugHunter commented 3 years ago

Here are full logs, including domain: scrapy crawl phpBB -o posts.csv 2020-12-10 15:46:55 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: phpBB_scraper) 2020-12-10 15:46:55 [scrapy.utils.log] INFO: Versions: lxml 4.6.2.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)], pyOpenSSL 20.0.0 (OpenSSL 1.1.1i 8 Dec 2020), cryptography 3.3, Platform Windows-10-10.0.17763-SP0 2020-12-10 15:46:55 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor 2020-12-10 15:46:55 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'phpBB_scraper', 'DOWNLOAD_DELAY': 3.0, 'NEWSPIDER_MODULE': 'phpBB_scraper.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['phpBB_scraper.spiders'], 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, ' 'like Gecko) Chrome/58.0.3029.110 Safari/537.36 ' 'OPR/45.0.2552.888'} 2020-12-10 15:46:55 [scrapy.extensions.telnet] INFO: Telnet Password: 0e4b12fe8aabc7f0 2020-12-10 15:46:55 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats'] 2020-12-10 15:46:55 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-12-10 15:46:55 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2020-12-10 15:46:55 [scrapy.middleware] INFO: Enabled item pipelines: [] 2020-12-10 15:46:55 [scrapy.core.engine] INFO: Spider opened 2020-12-10 15:46:55 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:46:55 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2020-12-10 15:46:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/robots.txt> (referer: None) 2020-12-10 15:47:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/index.php> (referer: None) 2020-12-10 15:47:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/> (referer: None) 2020-12-10 15:47:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184> from <POST https://immortalchessforum.com/ucp.php?mode=login&sid=95bc88ee77d2d57b5aff8f944ba8bb1c> 2020-12-10 15:47:10 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://immortalchessforum.com/index.php?sid=5fb8aafdc59c1cf382becf148c062ea7> from <POST https://immortalchessforum.com/ucp.php?mode=login&sid=5527c94613d96749e4f6d7971530e690> 2020-12-10 15:47:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184> (referer: https://immortalchessforum.com/index.php) 2020-12-10 15:47:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/index.php?sid=5fb8aafdc59c1cf382becf148c062ea7> (referer: https://immortalchessforum.com/) 2020-12-10 15:47:18 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://immortalchessforum.com/viewforum.php?f=4> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates) 2020-12-10 15:47:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=4> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=136> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=21> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=20> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=60> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=23> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=18> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=17> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=16> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=15> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=11> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:47:55 [scrapy.extensions.logstats] INFO: Crawled 16 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:47:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=10> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:48:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=9> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:48:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=8> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:48:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=6> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:48:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=89> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:48:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=26> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 15:48:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=7> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 15:48:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=844> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 15:48:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=1248> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 15:48:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=20&t=4142> (referer: https://immortalchessforum.com/viewforum.php?f=20) 2020-12-10 15:48:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=60&t=2509> (referer: https://immortalchessforum.com/viewforum.php?f=60) 2020-12-10 15:48:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=2748> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 15:48:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=1274> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 15:48:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=177> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:48:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=861> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:48:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=364> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:48:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=96> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:48:55 [scrapy.extensions.logstats] INFO: Crawled 33 pages (at 17 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:48:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=2740> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:49:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=4491> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:49:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5681> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:49:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=6&t=9> (referer: https://immortalchessforum.com/viewforum.php?f=6) 2020-12-10 15:49:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=89&t=2565> (referer: https://immortalchessforum.com/viewforum.php?f=89) 2020-12-10 15:49:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=2348> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=2837> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=2918> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=3000> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=2705> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=3250> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=3254> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=3696> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=3722> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=3740> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=3757> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=3873> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:49:55 [scrapy.extensions.logstats] INFO: Crawled 50 pages (at 17 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:49:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=616> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=4262> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=4308> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=4309> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=4431> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=1319> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=4461> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=3633> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=4659> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=4960> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=5033> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=5229> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=26&t=5434> (referer: https://immortalchessforum.com/viewforum.php?f=26) 2020-12-10 15:50:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=6&t=121> (referer: https://immortalchessforum.com/viewforum.php?f=6) 2020-12-10 15:50:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=6&t=547> (referer: https://immortalchessforum.com/viewforum.php?f=6) 2020-12-10 15:50:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=6&t=2666> (referer: https://immortalchessforum.com/viewforum.php?f=6) 2020-12-10 15:50:55 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:50:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=6&t=2992> (referer: https://immortalchessforum.com/viewforum.php?f=6) 2020-12-10 15:51:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5660> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=2428> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5666> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5674> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5675> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=155> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5693> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=62> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5628> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5447> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5297> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=438> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5711> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5713> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5714> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:51:55 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:51:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5715> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:52:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5716> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:52:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5390> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:52:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5720> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:52:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5653> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:52:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5619> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:52:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5722> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:52:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=5726> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:52:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=8&t=583> (referer: https://immortalchessforum.com/viewforum.php?f=8) 2020-12-10 15:52:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5106> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:52:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5160> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:52:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5173> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:52:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=910> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:52:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5253> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:52:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5259> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:52:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5261> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:52:55 [scrapy.extensions.logstats] INFO: Crawled 98 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:52:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5291> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:52:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5260> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=3866> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5403> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=175> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=24> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5351> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5484> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=181> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=1799> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5677> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5680> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5682> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=5426> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=633> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=1049> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:53:55 [scrapy.extensions.logstats] INFO: Crawled 114 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:53:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=9&t=52> (referer: https://immortalchessforum.com/viewforum.php?f=9) 2020-12-10 15:54:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=269> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=3230> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=3625> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=3849> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=3898> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=3915> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=4063> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=4519> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=4055> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=3899> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=4570> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=4745> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=4760> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=268> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=4061> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:55 [scrapy.extensions.logstats] INFO: Crawled 130 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:54:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=4189> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:54:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=5130> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:55:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=167> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:55:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=3013> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:55:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=2030> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:55:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=170> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:55:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=2386> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:55:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=166> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:55:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=10&t=5559> (referer: https://immortalchessforum.com/viewforum.php?f=10) 2020-12-10 15:55:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=1061> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:55:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=2189> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:55:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=2347> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:55:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=1837> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:55:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=4015> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:55:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=2685> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:55:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=2707> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:55:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=4266> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:55:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=4357> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:55:55 [scrapy.extensions.logstats] INFO: Crawled 148 pages (at 18 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:55:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=4359> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:56:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=4421> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:56:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=4508> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:56:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=4599> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:56:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=4980> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:56:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=5242> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:56:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=790> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:56:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=11&t=4951> (referer: https://immortalchessforum.com/viewforum.php?f=11) 2020-12-10 15:56:32 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=399> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:56:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://immortalchessforum.com/viewtopic.php?f=15&t=399>: HTTP status code is not handled or not allowed 2020-12-10 15:56:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=477> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:56:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=1919> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:56:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=2325> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:56:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=3692> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:56:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4019> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:56:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4056> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:56:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4090> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:56:55 [scrapy.extensions.logstats] INFO: Crawled 164 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:57:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4101> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=3019> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4064> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4544> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4580> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4556> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=1442> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4121> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4489> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4317> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4091> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=1631> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=15&t=4900> (referer: https://immortalchessforum.com/viewforum.php?f=15) 2020-12-10 15:57:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=2318> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:57:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=671> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:57:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=831> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:57:55 [scrapy.extensions.logstats] INFO: Crawled 180 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:57:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=3910> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:58:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=3911> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:58:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=84> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:58:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=5155> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:58:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=707> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:58:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=51> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:58:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=1869> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:58:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=16&t=5600> (referer: https://immortalchessforum.com/viewforum.php?f=16) 2020-12-10 15:58:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=510> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:58:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=3271> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:58:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=27> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:58:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=47> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:58:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=3327> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:58:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=4858> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:58:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=335> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:58:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=5072> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:58:55 [scrapy.extensions.logstats] INFO: Crawled 196 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:58:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=4822> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=5178> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=5066> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=708> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=787> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=5069> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=4536> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=199> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=5492> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=89> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=5502> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=417> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=908> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=1116> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=171> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=17&t=173> (referer: https://immortalchessforum.com/viewforum.php?f=17) 2020-12-10 15:59:55 [scrapy.extensions.logstats] INFO: Crawled 212 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 15:59:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=2902> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 15:59:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5318> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5319> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5320> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5321> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5322> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5323> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=4807> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5433> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=1876> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=550> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=4921> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=200> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5563> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5648> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5388> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=4917> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=4920> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:00:55 [scrapy.extensions.logstats] INFO: Crawled 230 pages (at 18 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 16:00:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=4916> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:01:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5676> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:01:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5673> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:01:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5687> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:01:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5688> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:01:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=18&t=5652> (referer: https://immortalchessforum.com/viewforum.php?f=18) 2020-12-10 16:01:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=3762> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:01:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5665> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:01:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5671> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:01:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5672> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:01:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=1566> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:01:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5679> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:01:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5684> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:01:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5686> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:01:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5685> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:01:55 [scrapy.extensions.logstats] INFO: Crawled 245 pages (at 15 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 16:01:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5692> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5697> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5157> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5633> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5701> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5702> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5703> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=4047> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5706> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5710> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5340> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=4836> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5704> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=1020> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=23&t=5325> (referer: https://immortalchessforum.com/viewforum.php?f=23) 2020-12-10 16:02:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=60&t=2585> (referer: https://immortalchessforum.com/viewforum.php?f=60) 2020-12-10 16:02:55 [scrapy.extensions.logstats] INFO: Crawled 261 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 16:02:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=60&t=3881> (referer: https://immortalchessforum.com/viewforum.php?f=60) 2020-12-10 16:03:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=60&t=5691> (referer: https://immortalchessforum.com/viewforum.php?f=60) 2020-12-10 16:03:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=1871> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=3006> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=2258> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=4355> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=4953> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=2865> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=4466> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=5122> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=5247> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=21&t=5695> (referer: https://immortalchessforum.com/viewforum.php?f=21) 2020-12-10 16:03:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=845> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:03:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=846> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:03:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=875> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:03:55 [scrapy.extensions.logstats] INFO: Crawled 276 pages (at 15 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 16:03:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=1003> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=1051> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=1522> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=3530> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=1657> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=2083> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=2248> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=2895> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=1142> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=4975> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=843> (referer: https://immortalchessforum.com/viewforum.php?f=136) 2020-12-10 16:04:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=779> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:04:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=1163> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:04:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=1773> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:04:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=2478> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:04:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=2619> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:04:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=2665> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:04:55 [scrapy.extensions.logstats] INFO: Crawled 293 pages (at 17 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 16:04:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=4287> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=215> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=4538> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:10 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=4814> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=4711> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=4912> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=4983> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=5316> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=5508> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=5643> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=3347> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=13> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=517> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=841> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=842> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:55 [scrapy.extensions.logstats] INFO: Crawled 308 pages (at 15 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 16:05:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=136&t=849> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:05:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=2320> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:06:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=4&t=4611> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:06:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=5&t=5141> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:06:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5142> (referer: https://immortalchessforum.com/viewforum.php?f=4) 2020-12-10 16:06:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=25> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 16:06:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=146> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 16:06:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewforum.php?f=5> (referer: https://immortalchessforum.com/index.php?sid=e80a051707489173e51eaefabd341184) 2020-12-10 16:06:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5636> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:06:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5596> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:06:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=648> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:06:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5556> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:06:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5647> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:06:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5070> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:06:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5654> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:06:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5662> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:06:55 [scrapy.extensions.logstats] INFO: Crawled 324 pages (at 16 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 16:06:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5206> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:06:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5661> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:07:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5698> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:07:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5712> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:07:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5717> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:07:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5724> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:07:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5536> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:07:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=146&t=3351> (referer: https://immortalchessforum.com/viewforum.php?f=146) 2020-12-10 16:07:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=146&t=4057> (referer: https://immortalchessforum.com/viewforum.php?f=146) 2020-12-10 16:07:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=146&t=3965> (referer: https://immortalchessforum.com/viewforum.php?f=146) 2020-12-10 16:07:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=146&t=4053> (referer: https://immortalchessforum.com/viewforum.php?f=146) 2020-12-10 16:07:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=146&t=4316> (referer: https://immortalchessforum.com/viewforum.php?f=146) 2020-12-10 16:07:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=146&t=4555> (referer: https://immortalchessforum.com/viewforum.php?f=146) 2020-12-10 16:07:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=146&t=4847> (referer: https://immortalchessforum.com/viewforum.php?f=146) 2020-12-10 16:07:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=146&t=5659> (referer: https://immortalchessforum.com/viewforum.php?f=146) 2020-12-10 16:07:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5569> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:07:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=957> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:07:55 [scrapy.extensions.logstats] INFO: Crawled 341 pages (at 17 pages/min), scraped 0 items (at 0 items/min) 2020-12-10 16:07:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5601> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:08:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5606> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:08:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5611> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:08:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5615> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:08:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5617> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:08:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=5622> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:08:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=1626> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:08:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://immortalchessforum.com/viewtopic.php?f=25&t=3862> (referer: https://immortalchessforum.com/viewforum.php?f=25) 2020-12-10 16:08:22 [scrapy.core.engine] INFO: Closing spider (finished) 2020-12-10 16:08:22 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 192908, 'downloader/request_count': 351, 'downloader/request_method_count/GET': 349, 'downloader/request_method_count/POST': 2, 'downloader/response_bytes': 4892930, 'downloader/response_count': 351, 'downloader/response_status_count/200': 348, 'downloader/response_status_count/302': 2, 'downloader/response_status_count/404': 1, 'dupefilter/filtered': 172, 'elapsed_time_seconds': 1286.409725, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2020, 12, 10, 11, 8, 22, 384640), 'httperror/response_ignored_count': 1, 'httperror/response_ignored_status_count/404': 1, 'log_count/DEBUG': 352, 'log_count/INFO': 32, 'request_depth_max': 3, 'response_received_count': 349, 'robotstxt/request_count': 1, 'robotstxt/response_count': 1, 'robotstxt/response_status_count/200': 1, 'scheduler/dequeued': 350, 'scheduler/dequeued/memory': 350, 'scheduler/enqueued': 350, 'scheduler/enqueued/memory': 350, 'start_time': datetime.datetime(2020, 12, 10, 10, 46, 55, 974915)} 2020-12-10 16:08:22 [scrapy.core.engine] INFO: Spider closed (finished)

phpbb.py: ` import re import scrapy from bs4 import BeautifulSoup from scrapy.http import Request

TODO: Please provide values for the following variables

Domains only, no urls

ALLOWED_DOMAINS = ['immortalchessforum.com']

Starting urls

START_URLS = ['https://immortalchessforum.com/']

Is login required? True or False.

FORM_LOGIN = True

Login username

USERNAME = 'REMOVED'

Login password

PASSWORD = 'REMOVED'

Login url

LOGIN_URL = 'https://immortalchessforum.com/index.php'

class PhpbbSpider(scrapy.Spider):

name = 'phpBB'
allowed_domains = ALLOWED_DOMAINS
start_urls = START_URLS
form_login = FORM_LOGIN
if form_login is True:
    username = USERNAME
    password = PASSWORD
    login_url = LOGIN_URL
    start_urls.insert(0, login_url)

username_xpath = '//p[contains(@class, "author")]//a[contains(@class, "username")]//text()'
post_count_xpath = '//dd[@class="profile-posts" or not(@class)]//a/text()'
post_time_xpath = '//div[@class="postbody"]//time/@datetime|//div[@class="postbody"]//p[@class="author"]/text()[2]'
post_text_xpath = '//div[@class="postbody"]//div[@class="content"]'

def parse(self, response):
    # LOGIN TO PHPBB BOARD AND CALL AFTER_LOGIN
    if self.form_login:
        formxpath = '//*[contains(@action, "login")]'
        formdata = {'username': self.username, 'password': self.password}
        form_request = scrapy.FormRequest.from_response(
                response,
                formdata=formdata,
                formxpath=formxpath,
                callback=self.after_login,
                dont_click=False
        )
        yield form_request
    else:
        # REQUEST SUB-FORUM TITLE LINKS
        links = response.xpath('//a[@class="forumtitle"]/@href').extract()
        for link in links:
            yield scrapy.Request(response.urljoin(link), callback=self.parse_topics)

def after_login(self, response):
    # CHECK LOGIN SUCCESS BEFORE MAKING REQUESTS
    if b'authentication failed' in response.body:
        self.logger.error('Login failed.')
        return
    else:
        # REQUEST SUB-FORUM TITLE LINKS
        links = response.xpath('//a[@class="forumtitle"]/@href').extract()
        for link in links:
            yield scrapy.Request(response.urljoin(link), callback=self.parse_topics)

def parse_topics(self, response):
    # REQUEST TOPIC TITLE LINKS
    links = response.xpath('//a[@class="topictitle"]/@href').extract()
    for link in links:
        yield scrapy.Request(response.urljoin(link), callback=self.parse_posts)

    # IF NEXT PAGE EXISTS, FOLLOW
    next_link = response.xpath('//li[@class="next"]//a[@rel="next"]/@href').extract_first()
    if next_link:
        yield scrapy.Request(response.urljoin(next_link), callback=self.parse_topics)   

def clean_quote(self, string):
    # CLEAN HTML TAGS FROM POST TEXT, MARK QUOTES
    soup = BeautifulSoup(string, 'lxml')
    block_quotes = soup.find_all('blockquote')
    for i, quote in enumerate(block_quotes):
        block_quotes[i] = '<quote-%s>=%s' % (str(i + 1), quote.get_text())
    return ''.join(block_quotes).strip()

def clean_text(self, string):
    # CLEAN HTML TAGS FROM POST TEXT, MARK REPLIES TO QUOTES
    tags = ['blockquote']
    soup = BeautifulSoup(string, 'lxml')
    for tag in tags:
        for i, item in enumerate(soup.find_all(tag)):
            item.replaceWith('<reply-%s>=' % str(i + 1))
    return re.sub(r' +', r' ', soup.get_text()).strip()

def parse_posts(self, response):
    # COLLECT FORUM POST DATA
    usernames = response.xpath(self.username_xpath).extract()
    n = len(usernames)
    if n > 0:
        post_counts = response.xpath(self.post_count_xpath).extract() or (n * [''])
        post_times = response.xpath(self.post_time_xpath).extract() or (n * [''])
        post_texts = response.xpath(self.post_text_xpath).extract() or (n * [''])
        post_quotes = [self.clean_quote(s) for s in post_texts]
        post_texts = [self.clean_text(s) for s in post_texts]

        # YIELD POST DATA
        for i in range(n):
            yield {'Username': str(usernames[i]).strip(), 'PostCount': str(post_counts[i]).strip(),
                   'PostTime': str(post_times[i]).strip(), 'PostText': post_texts[i], 'QuoteText': post_quotes[i]}

    # CLICK THROUGH NEXT PAGE
    next_link = response.xpath('//li[@class="next"]//a[@rel="next"]/@href').extract_first()
    if next_link:
        yield scrapy.Request(response.urljoin(next_link), callback=self.parse_posts)

` (maybe scraper is not working due to updated phpbb forum or something else?!)

mknghaloq commented 3 years ago

Hello friend, The scrapper seems to fail on this website because of a customization that changes the name of the div class who holds the username. So, the scrapper can not find it. Open the file phpBB.py and change this line:

username_xpath = '//p[contains(@class, "author")]//a[contains(@class, "username")]//text()'

into this line:

username_xpath = '//p[contains(@class, "author")]//span[contains(@class, "username-coloured")]//text()'

Also, to let scrapper see the next page you change this line:

next_link = response.xpath('//li[@class="next"]//a[@rel="next"]/@href').extract_first()

into this line.

next_link = response.xpath('//li[@class="arrow next"]//a[@rel="next"]/@href').extract_first()

Make two of this change. One next_link is in the section for seeing the next page of forums, and one is in the section for seeing the next page of a thread.

Now the scrapper will work properly on this website. I tested it and it scrapped many posts. In general if it does not work on one forum, try to look at the forum page source and know if they changed the names of some stuff.

Sincerely, mknghaloq