FATAL ERROR: heap out of memory, code from docs crashes

[resession]

i am using ubuntu linux, using nodejs v20.1.0. i am running basic code from the helia/unixfs docs.

async function test(){
const {FsDatastore} = await import('datastore-fs')
const {FsBlockstore} = await import('blockstore-fs')
const helias = await import('helia')
const {unixfs} = await import('@helia/unixfs')
const { CID } = await import('multiformats/cid')

const helia = await helias.createHelia({
  blockstore: new FsBlockstore(require('path').join(__dirname, 'data')),
  datastore: new FsDatastore(require('path').join(__dirname, 'data'))
})

const fs = unixfs(helia)

for await (const entry of fs.addAll([{
  path: 'foo.txt',
  content: Uint8Array.from([0, 1, 2, 3])
}])) {
  console.info(entry)
}
return null
}

test().then(console.log).catch(console.error)

the script returns what it should, but the problem is that after a few minutes, the script crashes. the error says that it is out of memory.

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

this is rest of the error message.

1: 0xc83eb0 node::Abort() [node]
 2: 0xb698df  [node]
 3: 0xe9ea20 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0xe9ed07 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 5: 0x10b0375  [node]
 6: 0x10b0904 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0x10c77f4 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node]
 8: 0x10c800c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0x1120dcc v8::internal::MinorGCJob::Task::RunInternal() [node]
10: 0xcfb386  [node]
11: 0xcfdf3f node::PerIsolatePlatformData::FlushForegroundTasksInternal() [node]
12: 0x185f1e6  [node]
13: 0x187164d  [node]
14: 0x185fb8e uv_run [node]
15: 0xba3ba6 node::SpinEventLoopInternal(node::Environment*) [node]
16: 0xcd1476 node::NodeMainInstance::Run() [node]
17: 0xc3d19f node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResultImpl const*) [node]
18: 0xc40542 node::Start(int, char**) [node]
19: 0x7f0050229d90  [/lib/x86_64-linux-gnu/libc.so.6]
20: 0x7f0050229e40 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
21: 0xba12ae _start [node]
Aborted (core dumped)

am i doing something wrong here, or did i use the wrong options? any ideas would be great.

[achingbrain]

I cannot replicate this.

Does this happen with LTS node? Do you have a more complete example?

[resession]

I cannot replicate this.

Does this happen with LTS node? Do you have a more complete example?

i tried it with lts 18.16.0 and i get the exact same error/crash after 1 or 2 minutes. i am using the same example, straight from the documentation.

[resession]

i'll try this on my windows system and see if i get the same error.

what version of node are you using?

[resession]

i ran the same exact code i my windows system, i get the following error on windows 10, node 18.10.0 and 20.2.0

again running the same code, this is from the docs

https://ipfs.github.io/helia-unixfs/modules/_helia_unixfs.html

TypeError: Cannot read properties of undefined (reading 'put')
    at persist (file:///D:/Proj/test/node_modules/ipfs-unixfs-importer/dist/src/utils/persist.js:10:22)
    at async file:///D:/Proj/test/node_modules/ipfs-unixfs-importer/dist/src/dag-builder/buffer-importer.js:31:29

[resession]

on my ubuntu linux system, using both node LTS 18.16.0 and node v20.2.0.

if i use just new new FsBlockstore(), by itself or new FsDatastore() by itself like

const helia = await helias.createHelia({
  blockstore: new FsBlockstore(require('path').join(__dirname, 'data'))
})

// or

const helia = await helias.createHelia({
  datastore: new FsDatastore(require('path').join(__dirname, 'data'))
})

then i do not get the heap out of memory error anymore. it tried it for a few minutes by themselves and no error was thrown and it did not crash.

problem is if i use them together

const helia = await helias.createHelia({
  blockstore: new FsBlockstore(require('path').join(__dirname, 'data')),
  datastore: new FsDatastore(require('path').join(__dirname, 'data'))
})

then the heap out of memory error is thrown and it crashes under 2 minutes.

[resession]

spoke too soon

using new FsBlockstore() or new FsDatastore() by itself works for a few minutes, longer, but still throws heap out of memory error and crashes

[achingbrain]

I've managed to replicate this, there is a fix in https://github.com/libp2p/js-libp2p/pull/1804 - thanks for your patience!