georgevak
commented
4 years ago same problem, let me ask you, my machine is 32 bit, i saw in terminal it says 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' '(KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}
does the fbcrawl runs on 32bit machines?
I am running fbcrawl from VMWare Windows 10 image. I have python 3.7, and I am seeing following error when I run the command:
2020-03-11 19:31:57 [fb] INFO: Going through the "save-device" checkpoint 2020-03-11 19:32:03 [fb] INFO: Scraping facebook page https://mbasic.facebook.com/cnn 2020-03-11 19:32:07 [scrapy.core.scraper] ERROR: Spider error processing <GET https://mbasic.facebook.com/cnn> (referer: https://mbasic.facebook.com/?_rdr) Traceback (most recent call last): File "c:\users\user\appdata\local\programs\python\python37\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks result = g.send(result) File "c:\users\user\appdata\local\programs\python\python37\lib\site-packages\scrapy\core\downloader\middleware.py", line 42, in process_request defer.returnValue((yield download_func(request=request, spider=spider))) File "c:\users\user\appdata\local\programs\python\python37\lib\site-packages\twisted\internet\defer.py", line 1362, in returnValue raise _DefGen_Return(val) twisted.internet.defer._DefGen_Return: <200 https://mbasic.facebook.com/cnn>
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "c:\users\user\appdata\local\programs\python\python37\lib\site-packages\scrapy\utils\defer.py", line 55, in mustbe_deferred result = f(*args, **kw) File "c:\users\user\appdata\local\programs\python\python37\lib\site-packages\scrapy\core\spidermw.py", line 60, in process_spider_input return scrape_func(response, request, spider) File "c:\users\user\appdata\local\programs\python\python37\lib\site-packages\scrapy\core\scraper.py", line 148, in call_spider warn_on_generator_with_return_value(spider, callback) File "c:\users\user\appdata\local\programs\python\python37\lib\site-packages\scrapy\utils\misc.py", line 202, in warn_on_generator_with_return_value if is_generator_with_return_value(callable): File "c:\users\user\appdata\local\programs\python\python37\lib\site-packages\scrapy\utils\misc.py", line 187, in is_generator_with_return_value tree = ast.parse(dedent(inspect.getsource(callable))) File "c:\users\user\appdata\local\programs\python\python37\lib\ast.py", line 35, in parse return compile(source, filename, mode, PyCF_ONLY_AST) File "", line 1
def parse_page(self, response):
^
IndentationError: unexpected indent
2020-03-11 19:32:07 [scrapy.core.engine] INFO: Closing spider (finished)