search

mythsman / hexo-douban

:cd: a Hexo plugin that helps you generate content for Douban books, movies, music, and games.
MIT License
455 stars 57 forks source link

爬取速度有点慢怎么破 #28

Closed dazkarieh closed 6 years ago

dazkarieh commented 6 years ago

INFO Start processing INFO 3683 movies have been loaded in 356577 ms INFO Generated: movies/index.html 整整6分钟，我以为没反应了，第一次直接跳出了

mythsman commented 6 years ago

额。。。你这东西太多了啊。。。三千多部呢。。。快不了了ヾ(°ｰ°ヾ)^?

dazkarieh commented 6 years ago

用hexo clean命令清空缓存后想再次生成页面，发现爬不了了，爬出来的是一个空页面。 xxxiMac:ian root# hexo douban -m INFO Start processing INFO 0 movies have been loaded in 1617 ms INFO Generated: movies/index.html xxxiMac:ian root# hexo clean INFO Deleted database. INFO Deleted public folder. xxxiMac:ian root# hexo douban -m INFO Start processing INFO 0 movies have been loaded in 1143 ms INFO Generated: movies/index.html xxxiMac:ian root#

mythsman commented 6 years ago

那可能是你一次性爬的太多了。。。ip给封了。。。

dazkarieh commented 6 years ago

早上又爬了一次，报错了 INFO 3683 movies have been loaded in 326620 ms Unhandled rejection TypeError: self.route.get(...)._data is not a function at names.forEach.name (/Users/dazkarieh/Public/ian/node_modules/hexo-douban/index.js:83:48) at Array.forEach () at /Users/dazkarieh/Public/ian/node_modules/hexo-douban/index.js:81:11 at tryCatcher (/Users/dazkarieh/Public/ian/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/Users/dazkarieh/Public/ian/node_modules/bluebird/js/release/promise.js:512:31) at Promise._settlePromise (/Users/dazkarieh/Public/ian/node_modules/bluebird/js/release/promise.js:569:18) at Promise._settlePromise0 (/Users/dazkarieh/Public/ian/node_modules/bluebird/js/release/promise.js:614:10) at Promise._settlePromises (/Users/dazkarieh/Public/ian/node_modules/bluebird/js/release/promise.js:693:18) at Async._drainQueue (/Users/dazkarieh/Public/ian/node_modules/bluebird/js/release/async.js:133:16) at Async._drainQueues (/Users/dazkarieh/Public/ian/node_modules/bluebird/js/release/async.js:143:10) at Immediate.Async.drainQueues [as _onImmediate] (/Users/dazkarieh/Public/ian/node_modules/bluebird/js/release/async.js:17:14) at runCallback (timers.js:696:18) at tryOnImmediate (timers.js:667:5) at processImmediate (timers.js:649:5) xxxiMac:ian root#

mythsman commented 6 years ago

这应该是个偶现的bug。。。过两天我处理下。。。

AlliotTech commented 6 years ago

感觉可以搞个增量爬取的 2333 三千多部 每次都爬 我感觉很慌。。。我这两百来部都感觉要爬一会儿。。。

mythsman commented 6 years ago

这个感觉不太好搞啊，，先关了吧，等我以后有想法了再弄弄看。。。