search

iMoe037 / scrapy-automotive

Scraped Car and Driver and other website for car data including make, models, specs, and pictures
0 stars 0 forks source link

Not getting any data returned? #2

Open alexmiller00 opened 7 years ago

alexmiller00 commented 7 years ago

I am not getting any data returned in my json file although i have used your command "scrapy crawl automotive -o mybetternameforafile.json" and it creates the file but it is empty? this is my shell output

2017-03-08 11:12:49 [scrapy.utils.log] INFO: Scrapy 1.3.2 started (bot: automotive) 2017-03-08 11:12:49 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'automotive.spiders', 'FEED_URI': 'mybetternameforafile.json', 'SPIDER_MODULES': ['automotive.spiders'], 'BOT_NAME': 'automotive', 'ROBOTSTXT_OBEY': True, 'FEED_FORMAT': 'json'} 2017-03-08 11:12:49 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.corestats.CoreStats'] 2017-03-08 11:12:49 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2017-03-08 11:12:49 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2017-03-08 11:12:49 [scrapy.middleware] INFO: Enabled item pipelines: ['automotive.pipelines.AutomotivePipeline'] 2017-03-08 11:12:49 [scrapy.core.engine] INFO: Spider opened 2017-03-08 11:12:49 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-03-08 11:12:49 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/robots.txt> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/bugatti> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/acura> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/alfa-romeo> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/bentley> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/buick> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/bmw> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/audi> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/chrysler> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/aston-martin> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/cadillac> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/dodge> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/chevrolet> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/ferrari> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/fiat> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/genesis> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/gmc> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/honda> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/infiniti> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/jaguar> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/hyundai> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/koenigsegg> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/lamborghini> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/ford> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/jeep> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/lexus> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/lincoln> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/kia> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/land-rover> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/lotus> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/maserati> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/mclaren> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/mercedes-amg> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/mercedes-benz> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/mercedes-maybach> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/mini> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/mazda> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/mitsubishi> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/nissan> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/pagani> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/porsche> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/ram> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/rolls-royce> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/spyker> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/subaru> (referer: None) 2017-03-08 11:12:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/smart> (referer: None) 2017-03-08 11:12:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/tesla> (referer: None) 2017-03-08 11:12:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/volvo> (referer: None) 2017-03-08 11:12:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/scion> (referer: None) 2017-03-08 11:12:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/volkswagen> (referer: None) 2017-03-08 11:12:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.caranddriver.com/toyota> (referer: None) 2017-03-08 11:12:50 [scrapy.core.engine] INFO: Closing spider (finished) 2017-03-08 11:12:50 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 11473, 'downloader/request_count': 51, 'downloader/request_method_count/GET': 51, 'downloader/response_bytes': 1285842, 'downloader/response_count': 51, 'downloader/response_status_count/200': 51, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2017, 3, 8, 10, 12, 50, 178655), 'log_count/DEBUG': 52, 'log_count/INFO': 7, 'response_received_count': 51, 'scheduler/dequeued': 50, 'scheduler/dequeued/memory': 50, 'scheduler/enqueued': 50, 'scheduler/enqueued/memory': 50, 'start_time': datetime.datetime(2017, 3, 8, 10, 12, 49, 239611)} 2017-03-08 11:12:50 [scrapy.core.engine] INFO: Spider closed (finished)

iMoe037 commented 7 years ago

Hey @alexmiller00

Unfortunately caranddriver went from static HTML to using Angular Components. Since they changed the structure of their HTML, the script no longer works. The project is not being actively maintained and it would require a bit of an overhaul and some time. If you need the data, send me your email and I'll send the files over.