JS heap out of memory

[miek770]

Hi again,

This is the latest hickup I got. As mentioned in the title, I'm not sure it's really a fussel issue but I document it here for the time being:

 --> Processing /mnt/Photos/110303 - Billard BPR/DSCF4562.JPG...
 ----> Copying to '/home/fussel/app/fussel/web/public/static/_gallery/albums/110303 - Billard BPR/original_DSCF4562.JPG'
 ------> Generating photo size... '/home/fussel/app/fussel/web/public/static/_gallery/albums/110303 - Billard BPR/500x661_DSCF4562.JPG'
 ------> Generating photo size... '/home/fussel/app/fussel/web/public/static/_gallery/albums/110303 - Billard BPR/800x1058_DSCF4562.JPG'
 ------> Generating photo size... '/home/fussel/app/fussel/web/public/static/_gallery/albums/110303 - Billard BPR/816x1080_DSCF4562.JPG'
~/app/fussel/web ~/app/fussel
yarn run v1.22.4
$ react-scripts build
Creating an optimized production build...

<--- Last few GCs --->

[3183:0x13b8998]  1936094 ms: Mark-sweep 613.1 (665.5) -> 612.9 (669.0) MB, 550.2 / 0.1 ms  allocation failure GC in old space requested
[3183:0x13b8998]  1936843 ms: Mark-sweep 612.9 (669.0) -> 612.8 (641.5) MB, 749.1 / 0.1 ms  last resort GC in old space requested
[3183:0x13b8998]  1937374 ms: Mark-sweep 612.8 (641.5) -> 612.8 (641.5) MB, 531.0 / 0.1 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x55311aa9 <JSObject>
    1: _send [internal/child_process.js:676] [bytecode=0x9ed466f1 offset=586](this=0x582f4935 <ChildProcess map = 0x2a2f1e41>,message=0x40fa5ad9 <JSArray[4]>,handle=0x55304125 <undefined>,options=0x40fa5b15 <Object map = 0x2a2f1ef1>,callback=0x55304125 <undefined>)
    2: send [internal/child_process.js:586] [bytecode=0x3349abdd offset=131](this=0x582f4935 <ChildProcess map = 0x2a2f1e41>,message=0x4...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Retried, got it again:

~/app/fussel/web ~/app/fussel
yarn run v1.22.4
$ react-scripts build
Creating an optimized production build...

<--- Last few GCs --->

[4074:0x134a998]  2173523 ms: Mark-sweep 682.5 (735.5) -> 682.4 (735.5) MB, 5834.5 / 0.0 ms  allocation failure GC in old space requested
[4074:0x134a998]  2181101 ms: Mark-sweep 682.4 (735.5) -> 682.4 (720.5) MB, 5880.3 / 0.0 ms  last resort GC in old space requested
[4074:0x134a998]  2187989 ms: Mark-sweep 682.4 (720.5) -> 682.4 (720.5) MB, 6888.0 / 0.2 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x5b991aa9 <JSObject>
    1: visitQueue [/home/fussel/app/fussel/web/node_modules/@babel/traverse/lib/context.js:~90] [pc=0x2fc073fc](this=0x307fc195 <TraversalContext map = 0x294fd0d5>,queue=0x307fc1dd <JSArray[1]>)
    2: visitSingle [/home/fussel/app/fussel/web/node_modules/@babel/traverse/lib/context.js:~82] [pc=0x2fc13d1c](this=0x307fc195 <TraversalContext map = 0x294fd0d5>,node=0x31ba4e35 <Node map = 0x4af182c1>,ke...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

From here, it looks like --max-old-space-size should be increased to allow more memory usage. I have 2GB of RAM plus 2GB of swap, but the latter could easily be increased to whatever I need to build the site. I'm trying to find where to up this limit in the code but might not be quick about it.

[cbenning]

Interesting, I have never encountered this. This looks like a yarn / node thing. I wonder if it is a result of your huge library somehow?

[cbenning]

I see you're running yarn run and not yarn build. Do you get the same thing with yarn build?

[cbenning]

Can you give this a try? from within the web folder?

export NODE_OPTIONS="--max-old-space-size=4096" yarn build

[miek770]

Interesting, I have never encountered this. This looks like a yarn / node thing. I wonder if it is a result of your huge library somehow?

I'm sure it is! I might have to break it up in a few sublibrairies, but for now I'll keep trying.

I see you're running yarn run and not yarn build. Do you get the same thing with yarn build?

No it's just how yarn reports it, I'm simply running generate_site.sh:

(app) fussel@odroid:~/app/fussel$ cat generate_site.sh
#!/usr/bin/env bash
set -e

SCRIPT_PATH="${BASH_SOURCE[0]}"

CURRENT_PATH=$(dirname $SCRIPT_PATH)
cd $CURRENT_PATH

# Generate site
./fussel.py

# Rebuild node site
pushd web
yarn build
popd

printf "Site generated at: ${CURRENT_PATH}/web/build"
printf "\n\n To validate build run: \n   python -m http.server --directory web/build\n\n"

(app) fussel@odroid:~/app/fussel$ pushd web
~/app/fussel/web ~/app/fussel
(app) fussel@odroid:~/app/fussel/web$ NODE_OPTIONS="--max-old-space-size=8192" yarn build
yarn run v1.22.4
$ react-scripts build

I'm running it again with NODE_OPTIONS="--max-old-space-size=2048" yarn build from here instead of your suggestion (it didn't seem to do anything, the command just exited with no output). I'll see if it works or if more memory is required. It's weird, I got the impression that it didn't want to use my swap and just failed when called with anything bigger than 2048.

From here I see the max memory in node.js is set to 512 MB by default.

Last bit of info until the command finishes:

michel@odroid:/home/fussel/app/fussel/web$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           1.9G        678M         46M        106M        1.2G        1.1G
Swap:          2.0G        210M        1.8G

[miek770]

Yay it worked! I probably wouldn't change the script but would add something about this issue in the readme's installation procedure.

[cbenning]

My thoughts exactly. Thanks a lot for testing this!

[miek770]

You're welcome, thanks for the great app! Please feel free to close this issue when you want.

[cbenning]

Update the FAQ in the README in v2