Unable to build project with 3box - vue-cli - Allocation failed - JavaScript heap out of memory

[jamesmorgan]

Describe the bug When building a project with 3Box in via vue-cli - I receive a Allocation failed - JavaScript heap out of memory

To Reproduce Steps to reproduce the behavior:

1. Install vue-cli
2. Run vue create and create a simple project
3. Install 3box
4. run npm run build
5. See out of memory error

Expected behavior Project builds as normal

Desktop (please complete the following information):

• OS: Mac OSX
• Node: v8.9.4
• Npe: 6.9.0
• vue-cli: 3.7.0

Additional context Console output below:

I have also tried bumping memory opts to 4gb

npm run build

> futball-cards-vue@0.1.0 build /Users/jamesmorgan/Dropbox/workspace-blockrocket/nifty-football-frontend
> npx --max_old_space_size=4096 vue-cli-service build

⠋  Building for production...
<--- Last few GCs --->

[24906:0x102801000]   272354 ms: Mark-sweep 1413.1 (1525.4) -> 1413.1 (1525.4) MB, 1148.9 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1149 ms) last resort GC in old space requested
[24906:0x102801000]   273494 ms: Mark-sweep 1413.1 (1525.4) -> 1413.1 (1525.4) MB, 1138.8 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x16d218025ee1 <JSObject>
    1: SourceMapGenerator_addMapping [/Users/jamesmorgan/Dropbox/workspace-blockrocket/nifty-football-frontend/node_modules/terser/node_modules/source-map/lib/source-map-generator.js:~103] [pc=0x169f074c2ef5](this=0x16d2b7ee8ab1 <SourceMapGenerator map = 0x16d2c1ecfe61>,aArgs=0x16d23f905231 <Object map = 0x16d283e69a89>)
    2: add [/Users/jamesmorgan/Dropbox/workspace-blockrocket/nifty-football...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 4: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 5: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 6: 0x169f0738463d
⠙  Building for production...
<--- Last few GCs --->

[24908:0x103800000]   455631 ms: Mark-sweep 1411.6 (1505.9) -> 1411.6 (1502.9) MB, 1146.5 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1147 ms) last resort GC in old space requested
[24908:0x103800000]   456791 ms: Mark-sweep 1411.6 (1502.9) -> 1411.6 (1502.9) MB, 1159.8 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2f2d01d25ee1 <JSObject>
    1: SourceMapGenerator_addMapping [/Users/jamesmorgan/Dropbox/workspace-blockrocket/nifty-football-frontend/node_modules/terser/node_modules/source-map/lib/source-map-generator.js:~103] [pc=0x280d740558b](this=0x2f2d96dd3fc1 <SourceMapGenerator map = 0x2f2da465d9b1>,aArgs=0x2f2df226e221 <Object map = 0x2f2da465fb61>)
    2: add [/Users/jamesmorgan/Dropbox/workspace-blockrocket/nifty-football-...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 4: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 5: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/jamesmorgan/.nvm/versions/node/v8.9.4/bin/node]
 6: 0x280d6b0463d

[zachferland]

I could run with the following, to increase node memory limit

NODE_OPTIONS=--max_old_space_size=4096 npm run build

but something not playing nice still with the default webpack plugins used in vue, the memory footprint should not be that much and other webpack builds are not hitting the node limit. The size of the lib is likely not the cause, its due to some other process running, could be transpiling or linting or other pre processors, can't say for sure at moment. This temp fix may also be slow to build.

Also looks like the production configs end up with a much bigger build than when we build it with webpack

[jamesmorgan]

Ye this seems to fix it, thanks guys

[Schwartz10]

@zachferland @oed I'm running into the same problem. This likely due to main pointing to the lib directory, which does not bundle ipfs or orbit-db as dependencies:

When a client application is running a bundling process (like webpack), it reads those require statements and attempts to bundle them too, using a huge amount of memory. Additionally, ipfs is built for node and not the browser, so the bundler will go through ipfs' dependency tree and try to bundle those deps too (which have large crypto libraries). Any application that is using syntax like:

import Box from '3box'

will be using a lot of memory to bundle 3box.

I'm not sure what the best solution is. At first I thought one easy solution is to change the entry to point to the dist directory - less memory is used because we don't need to bundle any of 3box's dependencies. For some reason this didn't work in my build tho.

[michaelsena]

Thanks for the recommendations @Schwartz10! In addition to solving this long term with one of the solutions you proposed, we also have a short term solve for this sprint to document (in the readme or elsewhere perhaps) how to overcome this issue by increasing the memory allocation for the build process.

[Schwartz10]

Ok great @michaelsena thank you! Also, not sure if you saw but i edited some of my original post after doing some testing with an idea i initially thought would work.

Unfortunately adding to the memory allocation isn't solving the problem for me because 3box code is getting bundled inside an app that already uses a ton of memory to bundle. I'm wondering, is there another way around this? Why not compile ipfs and orbit-db (and any other deps) into the bundle? There might be another short term solution that fixes the problem without allocating more memory.

[zachferland]

thanks the followup @Schwartz10! The lib folder is just es5, for compatibility, no bundling, which is intentional. This is mostly so that you can still build it how you want, build node/web, shim libs specific to your env, avoid conflicts with other preprocessors, allow optimizations like tree shaking. (ie. if someone already used IPFS and then also used the dist, they would bundle ipfs twice).

I would be more concerned if we export the dist by default (that is mostly just for people to fetch from CDN if they want, or in worst case can't build it themselves), since all the above would be lost.

But yeah ipfs is large, and could alone be contributing to it. It also a combined with the effect of what other tooling, parser, preprocessors etc, you have in your build. For example we can build it without hitting any limits with the webpack config in this repo. Do you just have a webpack build? can you bundle/build just Orbitdb or ipfs in it? But could be possible the es5 output, or if anything else in there, could be giving it trouble to. You could also try build from importing/requiring the src folder and see if that works.

Im still interested to figure out exactly what it is. Surprised the memory option didn't work, although not great may be able to increase further even.

[Schwartz10]

The lib folder is just es5, for compatibility, no bundling, which is intentional. This is mostly so that you can still build it how you want, build node/web, shim libs specific to your env, avoid conflicts with other preprocessors, allow optimizations like tree shaking. (ie. if someone already used IPFS and then also used the dist, they would bundle ipfs twice).

Ah i see. Makes a lot of sense.

You could also try build from importing/requiring the src folder and see if that works.

This is a good idea. I spent the majority of the day yesterday figuring out what was causing the memory issue (a friend found this issue after we had figured out what was causing it lol). So I'll have another crack at it today/tomorrow and see if I can figure out how to make things work.

[0xjjpa]

Joining the conversation as we are having similar issues. Right now we could work around the bundle issue with the light-weight version you guys put together, but since we are looking to use the Threads API soon enough, I'm sure that won't be a viable solution anymore.

My team will poke at some ideas and workarounds, in the meantime, I'll be in the Discord channel (tag:jjperezaguinaga) in case you have any suggestions. Thanks!

[oed]

Thanks for chiming in @jjperezaguinaga! Happy to hear any ideas your team has 🙂

The new version with updated threads is being released tomorrow and will be a significant reduction in size. Hopefully that will mitigate this issue for most people.

[0xjjpa]

me hoping tomorrow is already today in my timezone

Jokes aside, regarding ideas we were pretty much looking into exploring the dist version of 3Box to see if we can avoid building it ourselves and instead defer it on page load. Wouldn't be the cleanest solution but it's worth a try.

Regarding the IPFS issue, I explored using js-ipfs with service workers at some point instead of having to bundle IPFS directly within an app. The idea is that in the same way we have MetaMask in order to be our web3 provider, we could use something else as a IPFS provider. Either a chrome extension (seems like a bad idea ux-wise) or just a lib that works as an iframe and behind the scenes encapsulates js-ipfs calls to allow pinning directly on the web instead of using a node. Never got all the way with that, but might be worth exploring in the future.

[oed]

@jjperezaguinaga That's definitely a great idea :) We actually explored it as well and for a short period of time 3box-js shipped without ipfs and instead used ipfs though an iframe as you describe. Unfortunately we had to roll that back because we where getting strange bugs. The good thing about it is that data can be shared between dapps so every dapp doesn't need to sync data from scratch.

It turned out that the bugs where not because of that we had ipfs within the iframe, but because we had the orbitdb cache (where the latest heads of DBs are stored) also in the iframe and we had a bug in that code. We have a fix for this now so it's definitely something to get back to soon!

Another thing to consider is security. We don't want any random dapp to start messing around with the data in the iframe such that it breaks the users data store.

[Schwartz10]

@oed @zachferland just bundled using only babel and no bundling software (webpack, rollup..etc). However when I try to link to this pacakge locally i get:

 aragon-profile (babel) $ npm link
npm ERR! path /Users/jonathanschwartz/Documents/owl/aragon-profile/node_modules/orbit-db
npm ERR! code EISGIT
npm ERR! git /Users/jonathanschwartz/Documents/owl/aragon-profile/node_modules/orbit-db: Appears to be a git repo or submodule.
npm ERR! git     /Users/jonathanschwartz/Documents/owl/aragon-profile/node_modules/orbit-db
npm ERR! git Refusing to remove it. Update manually,
npm ERR! git or move it out of the way first.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/jonathanschwartz/.npm/_logs/2019-06-12T17_16_43_710Z-debug.log

I went into the orbit-db node module in this directory, removed the .git directory and npm linking worked. Not sure what orbit did on their end to make this happen?

Also as a side node, when i ls -a into orbit as a node module, i get a bunch of files that I probably don't need:

 orbit-db (babel) $ ls -a
.                       API.md                  CONTRIBUTORS.md         Makefile                dist                    src
..                      CHANGELOG.md            FAQ.md                  README.md               docker                  test
.circleci               CODE_OF_CONDUCT.md      GUIDE.md                benchmarks              ipfs
.git                    CONTRIBUTING.md         LICENSE                 conf                    package.json

[oed]

Oh! Sounds like a problem with the npm release from orbit-db. Maybe they accidentally added .git to their package? I'll ping them and ask!

[Schwartz10]

Think that's definitely the issue. Please keep me updated on this!

[shamb0t]

Hey @Schwartz10 @oed thanks for the heads up! Just published an orbit-db patch without the .git dir. Does this fix the issue?

[kfth]

worked for me: NODE_OPTIONS=--max_old_space_size=4096 npm run build