kailu3
commented
4 years ago Hey! I am also getting the same result. My outputs are empty as well. I think you are correct: Airbnb probably changed some code on their website. I guess that's the thing about web-scrapers.. They're impossible to maintain since websites are constantly changing. I would start from first principles. See if you can scrape anything from a link like this one
and go from there. Let me know what you end up getting! :) Sorry for the late response. I realized I don't get notifications from these.
Hi, I am trying to rebuild your scraper. I was very happy to find something like your approach combining scrapy and splash as it seems to be the only way to get hold of the JSON database. Many thanks for this!! Have you run your code recently? I don't see any bad HTML errors coming back. Docker is up and running but my output is empty. Maybe Airbnb has adapted the code recently. Cheers marcello
2020-04-17 23:51:22 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: airbnb_scraper) 2020-04-17 23:51:22 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.6 (default, Jan 8 2020, 13:42:34) - [Clang 4.0.1 (tags/RELEASE_401/final)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Darwin-19.0.0-x86_64-i386-64bit 2020-04-17 23:51:22 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'airbnb_scraper', 'CONCURRENT_REQUESTS_PER_DOMAIN': 16, 'CONCURRENT_REQUESTS_PER_IP': 16, 'DOWNLOAD_DELAY': 20, 'FEED_FORMAT': 'json', 'FEED_URI': 'Berlin.json', 'NEWSPIDER_MODULE': 'airbnb_scraper.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['airbnb_scraper.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1'} 2020-04-17 23:51:22 [scrapy.extensions.telnet] INFO: Telnet Password: e654a180743fce0d 2020-04-17 23:51:22 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats'] 2020-04-17 23:51:22 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_splash.SplashCookiesMiddleware', 'scrapy_splash.SplashMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-04-17 23:51:22 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy_splash.SplashDeduplicateArgsMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2020-04-17 23:51:22 [scrapy.middleware] INFO: Enabled item pipelines: [] 2020-04-17 23:51:22 [scrapy.core.engine] INFO: Spider opened 2020-04-17 23:51:22 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2020-04-17 23:51:22 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2020-04-17 23:51:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.airbnb.com/robots.txt> (referer: None) 2020-04-17 23:51:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.airbnb.com/api/v2/explore_tabs?_format=for_explore_search_web&_intents=p1&allow_override%5B%5D=&auto_ib=false&client_session_id=621cf853-d03e-4108-b717-c14962b6ab8b¤cy=CAD&experiences_per_grid=20&fetch_filters=true&guidebooks_per_grid=20&has_zero_guest_treatment=true&is_guided_search=true&is_new_cards_experiment=true&is_standard_search=true&items_per_grid=18&key=d306zoyjsyarp7ifhu67rjxn52tv0t20&locale=en&luxury_pre_launch=false&metadata_only=false&query=Berlin&query_understanding_enabled=true&refinement_paths%5B%5D=%2Fhomes&s_tag=QLb9RB7g&search_type=FILTER_CHANGE&selected_tab_id=home_tab&show_groupings=true&supports_for_you_v3=true&timezone_offset=-240&version=1.5.6&price_min=100&price_max=200> (referer: None) 2020-04-17 23:51:24 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.airbnb.com/api/v2/explore_tabs?_format=for_explore_search_web&_intents=p1&allow_override%5B%5D=&auto_ib=false&client_session_id=621cf853-d03e-4108-b717-c14962b6ab8b¤cy=CAD&experiences_per_grid=20&fetch_filters=true&guidebooks_per_grid=20&has_zero_guest_treatment=true&is_guided_search=true&is_new_cards_experiment=true&is_standard_search=true&items_per_grid=18&key=d306zoyjsyarp7ifhu67rjxn52tv0t20&locale=en&luxury_pre_launch=false&metadata_only=false&query=Berlin&query_understanding_enabled=true&refinement_paths%5B%5D=%2Fhomes&s_tag=QLb9RB7g&search_type=FILTER_CHANGE&selected_tab_id=home_tab&show_groupings=true&supports_for_you_v3=true&timezone_offset=-240&version=1.5.6&price_min=100&price_max=200> (referer: None) Traceback (most recent call last): File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback yield next(it) File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy_splash/middleware.py", line 156, in process_spider_output for el in result: File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output for x in result: File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr> return (_set_referer(r) for r in result or ()) File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr> return (r for r in result or () if _filter(r)) File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr> return (r for r in result or () if _filter(r)) File "/Volumes/Elements Marcel Wieting/airbnb3/airbnb_scraper/spiders/airbnb.py", line 105, in parse_id data_dict[room_id]['price'] = home.get('pricing_quote').get('rate').get('amount') AttributeError: 'NoneType' object has no attribute 'get' 2020-04-17 23:51:24 [scrapy.core.engine] INFO: Closing spider (finished) 2020-04-17 23:51:24 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 1356, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 34990, 'downloader/response_count': 2, 'downloader/response_status_count/200': 2, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2020, 4, 17, 21, 51, 24, 9263), 'log_count/DEBUG': 2, 'log_count/ERROR': 1, 'log_count/INFO': 9, 'memusage/max': 66531328, 'memusage/startup': 66531328, 'response_received_count': 2, 'robotstxt/request_count': 1, 'robotstxt/response_count': 1, 'robotstxt/response_status_count/200': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'spider_exceptions/AttributeError': 1, 'start_time': datetime.datetime(2020, 4, 17, 21, 51, 22, 989936)} 2020-04-17 23:51:24 [scrapy.core.engine] INFO: Spider closed (finished) (base) MacBook-Pro:airbnb_scraper marcelwieting$