search

soulteary / hexo-blog

test 1000+ posts with hexo (see issue#1)
GNU General Public License v3.0
0 stars 0 forks source link

how to use hexo generate more than 1000+ posts #1

Closed soulteary closed 8 years ago

soulteary commented 8 years ago

I see someone met this problem again, so i write this.

before generate, you should init your hexo website.

git clone --branch master --depth 1 --single-branch https://github.com/soulteary/hexo-blog.git
cd hexo-blog
./bin/install.sh  --use-cnpm  ## npm mirror for china

prepare you posts, eg.:

wget https://github.com/soulteary/My-Blog-Posts/archive/master.zip
unzip master.zip -d posts/_posts && rm master.zip

edit node bin parms.

use this instead of #!/usr/bin/env node

#!/usr/bin/env node --max_old_space_size=4096 --optimize_for_size --max_executable_size=4096 --stack_size=4096

just run hexo by node_modules

./node_modules/.bin/hexo g

WAIT...

image

hexo-blog git:(master) ✗ ls public 
2007       2009       2011       2013       2015       archives   fancybox   js         tags
2008       2010       2012       2014       2016       css        index.html page
JLKM commented 8 years ago

Not able to generate 1786 docs

Awesome idea to extend Hexo with a plugin for 1000+ docs websites. Really appreciate your commitment in this area.

Made a clone of your repo this morning - but still no luck.

Here is, how I followed your instructions:

1) $ git clone --branch master --depth 1 --single-branch https://github.com/soulteary/hexo-blog.git hexo-blog

2) $ cd hexo-blog

3) $ ./bin/install.sh Dropped --use-cnpm as I live in Denmark. Returned the following warnings, which I just accepted:

npm WARN deprecated swig@1.4.2: This package is no longer maintained
npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN prefer global marked@0.3.6 should be installed with -g

> dtrace-provider@0.6.0 install D:\vscode\hexo\hexo-blog\node_modules\dtrace-provider
> node scripts/install.js

> hexo-util@0.6.0 postinstall D:\vscode\hexo\hexo-blog\node_modules\hexo-util
> npm run build:highlight

> hexo-util@0.6.0 build:highlight D:\vscode\hexo\hexo-blog\node_modules\hexo-util
> node scripts/build_highlight_alias.js > highlight_alias.json

> hexo-extend-plugin@0.1.8 postinstall D:\vscode\hexo\hexo-blog\node_modules\hexo-extend-plugin
> patch/index.js --backup

4) Changed C:\Users\Name\AppData\Roaming\npm\node_modules\hexo-cli\bin\hexo to:

#!/usr/bin/env node --max_old_space_size=4096 --optimize_for_size --max_executable_size=4096 --stack_size=4096`

'use strict';

require('../lib/hexo')();

5) $ ./node_modules/.bin/hexo g with the aprox. 23 docs included in your sample. No errors during generation. No errors at localhost:4000

6) Replaced your posts with 1786 of my own in main _source-folder.

7) $ ./node_modules/.bin/hexo gwith this error:

Nyt forsøg med 1786 poster:
Jan@JLKM1 MINGW64 /d/vscode/hexo/hexo-blog (master)
$ ./node_modules/.bin/hexo g
INFO  Start processing
INFO  Files loaded in 1.93 min
[hexo-extend-plugin@generate] convert dir list done: _posts
FATAL ERROR: Scavenger: semi-space copy
 Allocation failed - process out of memory

8) Changed C:\Users\Name\AppData\Roaming\npm\node_modules\hexo-cli\bin\hexo to:

#!/usr/bin/env node --max_old_space_size=8192 --optimize_for_size --max_executable_size=8192 --stack_size=8192

'use strict';

require('../lib/hexo')();

Loading-time 1.77 - but same error as in 7 above.

9) Divided the posts into 3 subfolders with approx. 600 docs each. Did this twice - but same story. New error-message in the first run, though. Please look below.


Jan@JLKM1 MINGW64 /d/vscode/hexo/hexo-blog (master)
$ hexo clean
INFO  Deleted database.
INFO  Deleted public folder.

Jan@JLKM1 MINGW64 /d/vscode/hexo/hexo-blog (master)
$ ./node_modules/.bin/hexo g
INFO  Start processing
INFO  Files loaded in 2.32 min
[hexo-extend-plugin@generate] convert dir list done: _posts
FATAL ERROR: CALL\_AND\_RETRY\_LAST Allocation failed - JavaScript heap out of memo                                      ry

Public-folder is created - but with no docs in it.

Should I also use command hexo extend-features --on somehow?

My setup:

Name Version
Windows 10
RAM 16 GB
Node.js 6.3.1

Any suggestions where to go from here?

soulteary commented 8 years ago

@JLKM , i've never generate posts/(coding) on windows recently....

Hmmmm...

  1. Maybe you can try run hexo g twice without clean and dirs.
  2. This plugin is support use custom archive struct dirs and something else without generate, i think you can use #!/usr/bin/env node --max_old_space_size=4096 --optimize_for_size --max_executable_size=4096 --stack_size=4096 to improve your runtime and try to upgrade your node version. (i use v6.5+...)
  3. Do you like to have a try that use digitalocean or linode vps for generate posts in Ubuntu? (set huge swap for extend memory).
  4. Try hugo?

Good Luck.

JLKM commented 8 years ago

Thank you for your handful of advice. Unfortunately still no luck - at least with Hexo.

Here my comments one after one:

  1. Made a test on Linux. The Linux setup has only 1 GB of RAM. Hexo generate crashed with 857 documents (take a look belo). But hexo g made it to the end with 500. This is not at all enough documents, though.
Hexo/bin-setting:
#!/usr/bin/env node --max_old_space_size=768 --optimize_for_size --max_executable_size=768 --stack_size=768

857 docs
# hexo clean
INFO  Deleted database.
# hexo g
INFO  Start processing
INFO  Files loaded in 15 min

<--- Last few GCs --->

 4143980 ms: Scavenge 549.4 (716.6) -> 549.3 (716.6) MB, 1345.7 / 0 ms [allocation failure].
 4144034 ms: Scavenge 549.4 (716.6) -> 549.3 (716.6) MB, 50.1 / 0 ms [allocation failure].
 4144089 ms: Scavenge 549.3 (716.6) -> 549.3 (716.6) MB, 55.5 / 0 ms [allocation failure].
 4144121 ms: Scavenge 549.6 (716.6) -> 549.5 (716.6) MB, 6.2 / 0 ms [allocation failure].
 4144129 ms: Scavenge 549.6 (716.6) -> 549.6 (716.6) MB, 6.0 / 0 ms [allocation failure].

<--- JS stacktrace --->
Cannot get stack trace in GC.
FATAL ERROR: Scavenger: semi-space copy
 Allocation failed - process out of memory

Back to Windows:

  1. Testet double run of hexo g with 1500 docs - and no clean in between. Still no luck. See log below:
$ hexo g "node --stack_size=12288 --max-old-space-size=12288"
INFO  Start processing
INFO  Files loaded in 5.73 min
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

$ hexo g "node --stack_size=12288 --max-old-space-size=12288"
INFO  Start processing
INFO  Files loaded in 5.75 min
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
  1. Got your plugin up and running the html-generation. Both on Node 6.3.1 and the shining new 6.6.0. Same confusing result with 1800 docs (on 16 GB RAM) and this hexo-cli/bin-setting: #!/usr/bin/env node --max_old_space_size=8192 --optimize_for_size --max_executable_size=8192 --stack_size=8192

And these commands:

$ ./node_modules/.bin/hexo g

INFO  Start processing
INFO  Files loaded in 2.55 min
[hexo-extend-plugin@generate] convert dir list done: _posts
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

The rest:

  1. Yes I might give it a shot on a cloud service with lot of RAM. On the other hand. Not being able to generate a few thousand docs with 16 GB RAM seems basicaly wrong.
  2. Hugo sounds like an interesting alternative - all the more with it's built-in taxonomies. But Hexo still matches my needs better at this moment, so I prefer to stick with this one.

You mention in Hexo #2165, that you have been able to generate 1000+ docs with Hexo last month. So have I (but not so many more than that)!

The challenge is, that I need to crete Hexo-sites with 3000-10.000 docs. Have you been able to do that?

If yes: How did you succeed, and did you use Hexo or your own plugin?

soulteary commented 8 years ago 
Made a test on Linux. The Linux setup has only 1 GB of RAM.

Try to set a huge swap, only 1G RAM o is too small to 3000 posts..

I think you should give node.js more RAM resource through it's swap, and use node like this:

#!/usr/bin/env node --max_old_space_size=4096 --optimize_for_size --max_executable_size=4096 --stack_size=4096

maybe you need set RAM more than 8192+ ?

soulteary commented 8 years ago

@JLKM maybe there's another way to solve it, use hexo server instead of generate, and use nginx or redis for static resource cache. i've tried this for last month because hexo could not generate more than 1000+ posts as default.

JLKM commented 8 years ago

Still on square 1.

Thank you once again for your suggestions. Have tried them all - and some more. Still no luck.

Linux with 1 GB RAM, Node.js 6.0.0, Hexo 3.2.2
  1. Have made several tests with 1, 2, 4, 8 and 12 GB settings in hexo-cli\bin\hexo: #!/usr/bin/env node --max_old_space_size=4096 --optimize_for_size --max_executable_size=4096 --stack_size=4096 Result: A blinking cursor for many minutes. Not even this message:
INFO  Start processing
INFO  Files loaded in 2 min
  1. Done the same tests with teh same BG settings in the Hexo g-parameters in bash: # hexo g "node --max_old_space_size=12288 --optimize_for_size --max_executable_size=12288 --stack_size=12288"

No better result than this:

INFO  Start processing
INFO  Files loaded in 13 min

<--- Last few GCs --->

12401112 ms: Mark-sweep 352.4 (716.6) -> 339.5 (716.6) MB, 355340.7 / 0 ms [allocation failure] [GC in old space requested].
12850521 ms: Mark-sweep 339.5 (716.6) -> 339.4 (716.6) MB, 449406.7 / 19 ms [allocation failure] [GC in old space requested].
13282137 ms: Mark-sweep 339.4 (716.6) -> 339.3 (716.6) MB, 431553.4 / 26 ms [last resort gc].
13603690 ms: Mark-sweep 339.3 (716.6) -> 339.3 (716.6) MB, 321553.3 / 19 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x4eb78695 <JS Object>
    1: valueOf [native v8natives.js:~80] [pc=0x492f28a4] (this=0xf2b89565 <JS Function linkToHelper (SharedFunctionInfo 0x432bbb3d)>)
    6: createWrap [/share/CACHEDEV1_DATA/odense/node_modules/lodash/lodash.js:~5418] [pc=0xc2165fc4] (this=0x4eb8ba25 <JS Global Object>,func=0xf2b89565 <JS Fun                                                                      ction linkToHelper (SharedFunctionInfo 0x432bbb3d)>,bitmask=1,thisArg=0x27a1a1d9     <an Object with map 0x2185...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memo                                                                      ry
Aborted
  1. My Linux setup only runs with the default setting #!/usr/bin/env node in hexo-cli\bin\hexo and less than about 700 docs.
Windows 10, Node 6.6.0 with 16 GB RAM
$ ./node_modules/.bin/hexo g "node --stack_size=12288 --max-old-space-size=12288"
INFO  Start processing
INFO  Files loaded in 2 min
[hexo-extend-plugin@generate] convert dir list done: _posts
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

The depressing result can be seen in last line.

Conclusion so far

Hexo generation in Hexo-folder og node_modules-folder apparently makes no difference for me. And you now admit, that you haven't been able to generate more than about 1000 docs at a time in Hexo. So I guess, we're stuck in this field.

Options left:

  1. Hugo (but not so many well-designed themes to choose from)
  2. Node 7.X with better memory-handling
  3. Hexo-fix.
  4. Perhaps your latest advise on using hexo server instead of hexo generate, which I don't understand. Please clarify this.
JLKM commented 8 years ago

No solution for this in sight on Hexo at this time. So please close this issue. But the solution do exist elsewhere: Hugo: 11.980 docs in 4 minutes and 40 seconds.

xu-song commented 6 years ago 
 bad option: --max_executable_size=4096

max_executable_size has been removed since node 8