rugantio
commented
5 years ago Thank you for opening the issue. I programmed a new "date" attribute that makes sure that as soon as the first post older than "date" is crawled the spider is halted.
The "date" format is "%Y-%m-%d" such as "2018-05-22" and can be passed as a parameter at runtime, for example:
scrapy crawl fb -a email="XXXXXX@gmail.com" -a password="XXXXXXXXXXXXXXX" -a page="https://www.facebook.com/cocacolaGB/" -a date="2019-01-01" -o cocacola.csv The default behavior is to crawl everything up to the beginning.
Hello again. I have found that by setting the year I want the posts, the code does that year and also the previous one .. I just want to be scratched by 2019, but I also get 2018. Any way to fix it? Thank you.
scrapy crawl fb -a email="xxxxxxx@gmail.com" -a password="xxxxxxx." -a page="MotoShopOnline.es" -a year="2019" -a lang="en" -o moto.csv 2019-03-23 12:33:15 [scrapy.utils.log] INFO: Scrapy 1.5.2 started (bot: fbcrawl) 2019-03-23 12:33:16 [scrapy.utils.log] INFO: Versions: lxml 4.3.0.0, libxml2 2.9.9, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.6.7 (default, Oct 22 2018, 11:32:17) - [GCC 8.2.0], pyOpenSSL 19.0.0 (OpenSSL 1.1.1a 20 Nov 2018), cryptography 2.1.4, Platform Linux-4.15.0-46-generic-x86_64-with-Ubuntu-18.04-bionic 2019-03-23 12:33:16 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'fbcrawl', 'FEED_EXPORT_ENCODING': 'utf-8', 'FEED_EXPORT_FIELDS': ['source', 'shared_from', 'date', 'text', 'reactions', 'likes', 'ahah', 'love', 'wow', 'sigh', 'grrr', 'comments', 'url'], 'FEED_FORMAT': 'csv', 'FEED_URI': 'moto.csv', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'fbcrawl.spiders', 'SPIDER_MODULES': ['fbcrawl.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'} 2019-03-23 12:33:16 [scrapy.extensions.telnet] INFO: Telnet Password: e0f8938f46c492da 2019-03-23 12:33:17 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats'] 2019-03-23 12:33:17 [fb] INFO: Email and password provided, using these as credentials 2019-03-23 12:33:17 [fb] INFO: Page attribute provided, scraping "MotoShopOnline.es" 2019-03-23 12:33:17 [fb] INFO: Year attribute found, set scraping back to 2019 2019-03-23 12:33:17 [fb] INFO: Language attribute recognized, using "en" for the facebook interface 2019-03-23 12:33:18 [scrapy.core.engine] INFO: Spider opened 2019-03-23 12:33:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2019-03-23 12:33:20 [fb] INFO: Got stuck in "save-device" checkpoint 2019-03-23 12:33:20 [fb] INFO: I will now try to redirect to the correct page 2019-03-23 12:33:22 [fb] INFO: Scraping facebook page https://mbasic.facebook.com/MotoShopOnline.es 2019-03-23 12:33:23 [fb] INFO: Parsing post n = 0 2019-03-23 12:33:23 [fb] INFO: Parsing post n = 1 2019-03-23 12:33:24 [fb] INFO: Parsing post n = 2 2019-03-23 12:33:24 [fb] INFO: Parsing post n = 3 2019-03-23 12:33:24 [fb] INFO: Parsing post n = 4 2019-03-23 12:33:24 [fb] INFO: FLAG DOES NOT ALWAYS REPRESENT ACTUAL YEAR 2019-03-23 12:33:24 [fb] INFO: First page scraped, click on more! Flag not set, default flag = 2019 2019-03-23 12:33:26 [fb] INFO: Parsing post n = 5 2019-03-23 12:33:26 [fb] INFO: Parsing post n = 6 2019-03-23 12:33:26 [fb] INFO: Parsing post n = 7 2019-03-23 12:33:26 [fb] INFO: Parsing post n = 8 2019-03-23 12:33:26 [fb] INFO: Parsing post n = 9 2019-03-23 12:33:26 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:28 [fb] INFO: Parsing post n = 10 2019-03-23 12:33:28 [fb] INFO: Parsing post n = 11 2019-03-23 12:33:28 [fb] INFO: Parsing post n = 12 2019-03-23 12:33:28 [fb] INFO: Parsing post n = 13 2019-03-23 12:33:29 [fb] INFO: Parsing post n = 14 2019-03-23 12:33:29 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:30 [fb] INFO: Parsing post n = 15 2019-03-23 12:33:30 [fb] INFO: Parsing post n = 16 2019-03-23 12:33:30 [fb] INFO: Parsing post n = 17 2019-03-23 12:33:31 [fb] INFO: Parsing post n = 18 2019-03-23 12:33:31 [fb] INFO: Parsing post n = 19 2019-03-23 12:33:31 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:35 [fb] INFO: Parsing post n = 20 2019-03-23 12:33:35 [fb] INFO: Parsing post n = 21 2019-03-23 12:33:35 [fb] INFO: Parsing post n = 22 2019-03-23 12:33:35 [fb] INFO: Parsing post n = 23 2019-03-23 12:33:35 [fb] INFO: Parsing post n = 24 2019-03-23 12:33:36 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:37 [fb] INFO: Parsing post n = 25 2019-03-23 12:33:37 [fb] INFO: Parsing post n = 26 2019-03-23 12:33:38 [fb] INFO: Parsing post n = 27 2019-03-23 12:33:38 [fb] INFO: Parsing post n = 28 2019-03-23 12:33:38 [fb] INFO: Parsing post n = 29 2019-03-23 12:33:38 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:39 [fb] INFO: Parsing post n = 30 2019-03-23 12:33:39 [fb] INFO: Parsing post n = 31 2019-03-23 12:33:39 [fb] INFO: Parsing post n = 32 2019-03-23 12:33:40 [fb] INFO: Parsing post n = 33 2019-03-23 12:33:40 [fb] INFO: Parsing post n = 34 2019-03-23 12:33:40 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:41 [fb] INFO: Parsing post n = 35 2019-03-23 12:33:42 [fb] INFO: Parsing post n = 36 2019-03-23 12:33:42 [fb] INFO: Parsing post n = 37 2019-03-23 12:33:42 [fb] INFO: Parsing post n = 38 2019-03-23 12:33:42 [fb] INFO: Parsing post n = 39 2019-03-23 12:33:42 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:43 [fb] INFO: Parsing post n = 40 2019-03-23 12:33:43 [fb] INFO: Parsing post n = 41 2019-03-23 12:33:43 [fb] INFO: Parsing post n = 42 2019-03-23 12:33:43 [fb] INFO: Parsing post n = 43 2019-03-23 12:33:44 [fb] INFO: Parsing post n = 44 2019-03-23 12:33:44 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:46 [fb] INFO: Parsing post n = 45 2019-03-23 12:33:46 [fb] INFO: Parsing post n = 46 2019-03-23 12:33:46 [fb] INFO: Parsing post n = 47 2019-03-23 12:33:46 [fb] INFO: Parsing post n = 48 2019-03-23 12:33:47 [fb] INFO: Parsing post n = 49 2019-03-23 12:33:47 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:49 [fb] INFO: Parsing post n = 50 2019-03-23 12:33:49 [fb] INFO: Parsing post n = 51 2019-03-23 12:33:49 [fb] INFO: Parsing post n = 52 2019-03-23 12:33:49 [fb] INFO: Parsing post n = 53 2019-03-23 12:33:49 [fb] INFO: Parsing post n = 54 2019-03-23 12:33:50 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:51 [fb] INFO: Parsing post n = 55 2019-03-23 12:33:51 [fb] INFO: Parsing post n = 56 2019-03-23 12:33:51 [fb] INFO: Parsing post n = 57 2019-03-23 12:33:51 [fb] INFO: Parsing post n = 58 2019-03-23 12:33:52 [fb] INFO: Parsing post n = 59 2019-03-23 12:33:52 [fb] INFO: Page scraped, click on more! flag = 2019 2019-03-23 12:33:53 [fb] INFO: Parsing post n = 60 2019-03-23 12:33:53 [fb] INFO: Parsing post n = 61 2019-03-23 12:33:54 [fb] INFO: There are no more, flag set at = 2019 2019-03-23 12:33:54 [fb] INFO: XPATH not found for year 2018 2019-03-23 12:33:54 [fb] INFO: Trying with previous year, flag=2018 2019-03-23 12:33:54 [fb] INFO: The previous year to crawl is less than the parameter year: 2018 < 2019 2019-03-23 12:33:54 [fb] INFO: This is not handled well, please re-run with -a year="2018" or less 2019-03-23 12:33:54 [fb] INFO: New page found with flag 2018 2019-03-23 12:33:54 [scrapy.core.scraper] ERROR: Spider error processing <GET https://mbasic.facebook.com/MotoShopOnline.es?sectionLoadingID=m_timeline_loading_div_1554101999_0_36_timeline_unit%3A1%3A00000000001536657425%3A04611686018427387904%3A09223372036854775746%3A04611686018427387904&unit_cursor=timeline_unit%3A1%3A00000000001536657425%3A04611686018427387904%3A09223372036854775746%3A04611686018427387904&timeend=1554101999×tart=0&tm=AQAJ6WfaA8LJagg-&refid=17> (referer: https://mbasic.facebook.com/MotoShopOnline.es?sectionLoadingID=m_timeline_loading_div_1554101999_0_36_timeline_unit%3A1%3A00000000001536661899%3A04611686018427387904%3A09223372036854775751%3A04611686018427387904&unit_cursor=timeline_unit%3A1%3A00000000001536661899%3A04611686018427387904%3A09223372036854775751%3A04611686018427387904&timeend=1554101999×tart=0&tm=AQAJ6WfaA8LJagg-&refid=17) Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/defer.py", line 102, in iter_errback yield next(it) File "/usr/local/lib/python3.6/dist-packages/scrapy/spidermiddlewares/offsite.py", line 30, in process_spider_output for x in result: File "/usr/local/lib/python3.6/dist-packages/scrapy/spidermiddlewares/referer.py", line 339, in
return (_set_referer(r) for r in result or ())
File "/usr/local/lib/python3.6/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in
return (r for r in result or () if _filter(r))
File "/usr/local/lib/python3.6/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in
return (r for r in result or () if _filter(r))
File "/media/compartida/master/marzo/virgen/fbcrawl-master/fbcrawl/spiders/fbcrawl.py", line 179, in parse_page
new_page = response.urljoin(new_page[0])
IndexError: list index out of range
2019-03-23 12:33:55 [scrapy.core.engine] INFO: Closing spider (finished)
2019-03-23 12:33:55 [scrapy.extensions.feedexport] INFO: Stored csv feed (62 items) in: moto.csv
2019-03-23 12:33:55 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 303245,
'downloader/request_count': 142,
'downloader/request_method_count/GET': 140,
'downloader/request_method_count/POST': 2,
'downloader/response_bytes': 1203167,
'downloader/response_count': 142,
'downloader/response_status_count/200': 140,
'downloader/response_status_count/302': 2,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 3, 23, 11, 33, 55, 506133),
'item_scraped_count': 62,
'log_count/ERROR': 1,
'log_count/INFO': 94,
'memusage/max': 50860032,
'memusage/startup': 50860032,
'request_depth_max': 17,
'response_received_count': 140,
'scheduler/dequeued': 142,
'scheduler/dequeued/memory': 142,
'scheduler/enqueued': 142,
'scheduler/enqueued/memory': 142,
'spider_exceptions/IndexError': 1,
'start_time': datetime.datetime(2019, 3, 23, 11, 33, 18, 222031)}
2019-03-23 12:33:55 [scrapy.core.engine] INFO: Spider closed (finished)