decidedlygray
commented
6 years ago You are trying to run:
python ./xsscrapy.py -u "http://www.****day.com/" -u 30But -u 30 is not valid. Try one of these:
python ./xsscrapy.py -u "http://www.****day.com/" -r 30Or:
python ./xsscrapy.py -u "http://www.****day.com/" -c 30
lacroutelacroute
$ cat requirements.txt Scrapy==1.1.0rc3 pybloom==1.1 requests beautifulsoup Twisted==16.6.0 w3lib lxml==3.6.4 six cssselect pyopenssl cryptography queuelib
xsscrapy not work $ python ./xsscrapy.py -u "http://www.****day.com/" -u 30 2018-01-07 10:53:18 [scrapy] INFO: Scrapy 1.1.0rc3 started (bot: xsscrapy) 2018-01-07 10:53:18 [scrapy] INFO: Overridden settings: {'COOKIES_DEBUG': True, 'NEWSPIDER_MODULE': 'xsscrapy.spiders', 'SPIDER_MODULES': ['xsscrapy.spiders'], 'DUPEFILTER_CLASS': 'xsscrapy.bloomfilters.BloomURLDupeFilter', 'CONCURRENT_REQUESTS': '30', 'BOT_NAME': 'xsscrapy', 'DOWNLOAD_DELAY': '0'} 2018-01-07 10:53:19 [scrapy] INFO: Enabled extensions: ['scrapy.extensions.logstats.LogStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.corestats.CoreStats'] 2018-01-07 10:53:19 [scrapy] INFO: Enabled downloader middlewares: ['xsscrapy.middlewares.InjectedDupeFilter', 'xsscrapy.middlewares.RandomUserAgentMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2018-01-07 10:53:19 [scrapy] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2018-01-07 10:53:19 [scrapy] INFO: Enabled item pipelines: ['xsscrapy.pipelines.XSSCharFinder'] 2018-01-07 10:53:19 [scrapy] INFO: Spider opened 2018-01-07 10:53:19 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2018-01-07 10:53:19 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 2018-01-07 10:53:19 [scrapy] ERROR: Error while obtaining start requests Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/scrapy/core/engine.py", line 126, in _next_request request = next(slot.start_requests) File "/home/fakessh/Bureau/xsscrapy/xsscrapy/spiders/xss_spider.py", line 115, in start_requests yield Request(url=self.start_urls[0]) File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/init.py", line 25, in init self._set_url(url) File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/init.py", line 57, in _set_url raise ValueError('Missing scheme in request url: %s' % self._url) ValueError: Missing scheme in request url: 30 2018-01-07 10:53:19 [scrapy] INFO: Closing spider (finished) 2018-01-07 10:53:19 [scrapy] INFO: Dumping Scrapy stats: {'finish_reason': 'finished', 'finish_time': datetime.datetime(2018, 1, 7, 9, 53, 19, 364388), 'log_count/DEBUG': 1, 'log_count/ERROR': 1, 'log_count/INFO': 7, 'start_time': datetime.datetime(2018, 1, 7, 9, 53, 19, 308870)} 2018-01-07 10:53:19 [scrapy] INFO: Spider closed (finished)