jasheppa5
commented
8 years ago Hey Alex,
I have a few ideas:
- Your PhantomJS instance needs to be updated. There were multiple issues with version < 2.0 and an update fixed the issue for most users. Run "phantomjs --version" from the command line. You should have 2.0 or higher. The latest version of Malspider installs an updated version of PhantomJS, so if this is your problem you can update the code or run install/install_dependencies.
- Some users have reported random issues with PhantomJS hanging up without error. This appears to be a somewhat common problem with PhantomJS and I'll have to find a way to handle that from Malspider. I don't think this is the case for you, though, since Malspider isn't able to scrape even the first page.
I think you have an outdated PhantomJS version. Let's start there. I was able to crawl the site in question just fine, minus a character encoding issue I'm working on...
Keep me updated. Let me know if updating PhantomJS works. If not we'll keep debugging.
-James
On Tue, Oct 18, 2016 at 4:02 AM, Alex Shatberashvili < notifications@github.com> wrote:
2016-10-13 00:00:19+0400 [scrapy] INFO: Scrapy 0.24.4 started (bot: full_domain) 2016-10-13 00:00:19+0400 [scrapy] INFO: Optional features available: ssl, http11, django 2016-10-13 00:00:19+0400 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'malspider.spiders', 'SPIDER_MODULES': ['malspider.spiders'], 'LOG_FILE': 'logs/malspider/full_domain/ 76f0156890b611e6959d005056ae7ab0.log', 'USER_AGENT': 'Mozilla/5.0 (Android; Tablet; rv:30.0) Gecko/30.0 Firefox/30.0', 'BOT_NAME': 'full_domain'} 2016-10-13 00:00:19+0400 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState 2016-10-13 00:00:19+0400 [scrapy] INFO: Enabled downloader middlewares: RandomUserAgentMiddleware, HttpAuthMiddleware, DownloadTimeoutMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 2016-10-13 00:00:19+0400 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, WebdriverSpiderMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 2016-10-13 00:00:19+0400 [scrapy] INFO: Enabled item pipelines: DuplicateFilterPipeline, WhitelistFilterPipeline, MySQLPipeline 2016-10-13 00:00:19+0400 [full_domain] INFO: Spider opened 2016-10-13 00:00:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:00:19+0400 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6026 2016-10-13 00:00:19+0400 [scrapy] DEBUG: Web service listening on 127.0.0.1:6083 2016-10-13 00:00:19+0400 [scrapy] DEBUG: Downloading https://www.des.gov.ge with webdriver 2016-10-13 00:01:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:02:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:03:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:04:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:05:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:06:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:07:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:08:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:09:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:10:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:11:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:12:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:13:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:14:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:15:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2016-10-13 00:16:19+0400 [full_domain] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
This goes on for 24 hours, at which point the task is sigkilled. Any idea why this keeps happening?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ciscocsirt/malspider/issues/13, or mute the thread https://github.com/notifications/unsubscribe-auth/AR0QEMy6cA5S_xCYkrQf3qInNjLF4Bmzks5q1H0PgaJpZM4KZfGq .
wrinkl3
This goes on for 24 hours, at which point the task is sigkilled. Seems to happen to all the domains I try to monitor, but not every time. Any idea why this keeps happening?