Hashing multiple files in parallel causes "WASM out of memory"

aleksey-hoffman commented 4 years ago

Good day @Daninet. My project hashes multiple files in parallel using hash-wasm. There's about 20 components and each one of them creates their own new hash-wasm instance and hashes a specified file. The problem is, when it hashes too many files (usually ~20) at the same time, I get the following error:

RangeError: WebAssembly.instantiate(): Out of memory: wasm memory

The hashing is triggered by a JS scroll event (component hashes specified file when it enters the viewport). It takes about 20-30 parallel calls (depending on the files sizes) to fill up the WASM memory and cause the error. Sometimes when I stop the hashing (stop scrolling the page) and then resume it after a few seconds it throws the error immediately as if the memory wasn't cleaned up at all and a single call is all it takes to cause the error. What I don't understand is why all the instances are interconnected and affect each other.

And the weird things is, even when it throws the error, it still finishes hashing the file properly.

Code

fileHasher.js

const { createXXHash64 } = require('hash-wasm')
const fs = require('fs')

class Filehasher {
  constructor (path) {
    this.state = { isCanceled: false }
    this.interval
    this.readStream
    this.path = path
  }
  cancel () {
    this.state.isCanceled = true
  }
  async gen () {
    try {
      const xxhash64 = await createXXHash64()
      return new Promise((resolve, reject) => {
        // xxhash64.init() // the results are the same with and without running init()
        this.readStream = fs.createReadStream(this.path)
          .on('data', data => xxhash64.update(data))
          .on('end', () => {
            resolve(xxhash64.digest('hex'))
          })
        this.interval = setInterval(() => {
          if (this.state.isCanceled) {
            clearInterval(this.interval)
            this.readStream.destroy()
            reject('canceled')
          }
        }, 10)
      })
    } 
    catch (error) {
      console.log('Filehasher', error)
    }
  }
}

module.exports = { Filehasher }

testFileHasher.js

const Filehasher = require('./fileHasher.js').Filehasher

async function test() {
  let hasher = new Filehasher('C:/test/test.jpg')
  // Generating a random file name for console.time()
  let rand = Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15)
  console.time(`TIME: file name: ${rand}`)
  let hash = await hasher.gen()
  console.timeEnd(`TIME: file name: ${rand}`)
  console.log('HASH:', hash)
}

setInterval(() => {
  test()
}, 10)

The project is based on Electron (Node.js+ Chromium) so initially I thought it was a Chromium bug, but then I ran it on Node.js in terminal, and got the same problem.

In this example I'm emulating multiple parallel jobs by hashing a single 20 MB file every 10 ms (in the real app every component creates a new hash-wasm instance and hashes different file of different size). It usually takes ~80 ms to hash this particular image once, but since it's called every 10 ms it doesn't have time to finish the job. As you can see in the console output below it exceeds the memory limits at about 20 parallel calls (the same thing happens in the real app). Notice how the hashing time grows and how all the instances are affecting each other.

Console output

TIME: file name: t6dwi6q4iqa9vhz3e05eoa: 186.477ms
HASH: 16d7104d28427ead
TIME: file name: 1n6974yqi8lxcvrw17kkz: 205.672ms
HASH: 16d7104d28427ead
TIME: file name: 21qqruayukc99var45zhxv: 286.718ms
HASH: 16d7104d28427ead
TIME: file name: 5rn3zv151q529awqjraikd: 375.536ms
HASH: 16d7104d28427ead
TIME: file name: n0l8akz5svcw4ok828tg8: 430.861ms
HASH: 16d7104d28427ead
TIME: file name: 31psu981yhtclunqmbax68: 471.69ms
HASH: 16d7104d28427ead
TIME: file name: stjqbb9rc49z92oztf1xr: 530.379ms
HASH: 16d7104d28427ead
TIME: file name: s7im9fzj9cpgkc7okou7lv: 589.838ms
HASH: 16d7104d28427ead
TIME: file name: 0liv7g7dacid106fry5rh4ue: 638.518ms
HASH: 16d7104d28427ead
TIME: file name: bcvciuahm1y2ed6wijfwr: 667.826ms
HASH: 16d7104d28427ead
TIME: file name: 4swzggf1watp2yyz4ilf7: 692.667ms
HASH: 16d7104d28427ead
TIME: file name: dmdwzzzrg745cgny7uqck: 717.181ms
HASH: 16d7104d28427ead
TIME: file name: v9pilvspzgnxsd18bq7dz: 729.266ms
HASH: 16d7104d28427ead
TIME: file name: hcbafyv7s8re77y94jwcxt: 758.71ms
HASH: 16d7104d28427ead
TIME: file name: lgzwf15hpxsuntwj3j2hr: 792.363ms
HASH: 16d7104d28427ead
TIME: file name: jb5phq1cqur8zaeug7fpgb: 818.249ms
HASH: 16d7104d28427ead
TIME: file name: 5zckaxil5dnot39duvutl7: 847.001ms
HASH: 16d7104d28427ead
TIME: file name: 5wor89u2pql22w8zrjni0o: 881.599ms
HASH: 16d7104d28427ead
TIME: file name: uhnyvwk7h4muicrphha45: 915.705ms
HASH: 16d7104d28427ead
Filehasher RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
    at E:\test\node_modules\hash-wasm\dist\index.umd.js:215:50
    at Generator.next (<anonymous>)
    at fulfilled (E:\test\node_modules\hash-wasm\dist\index.umd.js:25:62)
    at runMicrotasks (<anonymous>)
    at runNextTicks (internal/process/task_queues.js:58:5)
    at listOnTimeout (internal/timers.js:520:9)
    at processTimers (internal/timers.js:494:7)
TIME: file name: n34dwiz4nis0r7yb992dzkd: 25.566ms
HASH: undefined
Filehasher RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
    at E:\test\node_modules\hash-wasm\dist\index.umd.js:215:50
    at Generator.next (<anonymous>)
    at fulfilled (E:\test\node_modules\hash-wasm\dist\index.umd.js:25:62)
TIME: file name: xtnbxnd4indsl39hzz5anm: 17.682ms
HASH: undefined
Filehasher RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
    at E:\test\node_modules\hash-wasm\dist\index.umd.js:215:50
    at Generator.next (<anonymous>)
    at fulfilled (E:\test\node_modules\hash-wasm\dist\index.umd.js:25:62)
TIME: file name: e9bx838karafd90gcwvcvr: 18.854ms
HASH: undefined
TIME: file name: 37zuklk2l5aj5l1u976po: 1.006s
HASH: 16d7104d28427ead
Filehasher RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
    at E:\test\node_modules\hash-wasm\dist\index.umd.js:215:50
    at Generator.next (<anonymous>)
    at fulfilled (E:\test\node_modules\hash-wasm\dist\index.umd.js:25:62)
TIME: file name: s14k148uspj7l9br734tjf: 19.249ms
HASH: undefined
Filehasher RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
    at E:\test\node_modules\hash-wasm\dist\index.umd.js:215:50
    at Generator.next (<anonymous>)
    at fulfilled (E:\test\node_modules\hash-wasm\dist\index.umd.js:25:62)
TIME: file name: 4fhpx3t5untuup04e9lq18: 19.541ms
HASH: undefined
Filehasher RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
    at E:\test\node_modules\hash-wasm\dist\index.umd.js:215:50
    at Generator.next (<anonymous>)
    at fulfilled (E:\test\node_modules\hash-wasm\dist\index.umd.js:25:62)
TIME: file name: qsiir4x7ukhh204ml3vq: 18.706ms
HASH: undefined
Filehasher RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
    at E:\test\node_modules\hash-wasm\dist\index.umd.js:215:50
    at Generator.next (<anonymous>)
    at fulfilled (E:\test\node_modules\hash-wasm\dist\index.umd.js:25:62)
TIME: file name: netg1satgw4ffnek38p2n: 22.462ms
HASH: undefined
Filehasher RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
    at E:\test\node_modules\hash-wasm\dist\index.umd.js:215:50
    at Generator.next (<anonymous>)
    at fulfilled (E:\test\node_modules\hash-wasm\dist\index.umd.js:25:62)
TIME: file name: e7c17bzsd3yywkh8pfhj: 21.935ms
HASH: undefined
Filehasher RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
    at E:\test\node_modules\hash-wasm\dist\index.umd.js:215:50
    at Generator.next (<anonymous>)
    at fulfilled (E:\test\node_modules\hash-wasm\dist\index.umd.js:25:62)
TIME: file name: lbnqpty95gelr0nya4mgp: 19.821ms
HASH: undefined
TIME: file name: edynbbpaky5q9bk2haswjr: 1.158s
HASH: 16d7104d28427ead
...

Questions

Surprisingly this many parallel calls do not block the main JS thread even though it's very computationally intensive, but as soon as I get this WASM error, it starts blocking the main thread and UI starts freezing.

I tried creating a separate web worker for each computation but it takes up a lot of RAM and crashes the app at 20+ parallel computations.

Is there something I'm doing wrong?
Is there a way to force hash-wasm to wait until all the other instances are done so it doesn't exceed the WASM memory limits?
I'm not familiar with WASM, is it possible to allocate a separate isolated WASM instance with its own memory for each hash-wasm instance so they don't share the same memory limit? Would it overwhelm and block the main JS thread if it were to run on multiple WASM instances?

Environment: OS: Win10 x64 hash-wasm: v4.1.0 Exec env: the same results everywhere:

Node.js v14.5.0;
Electron renderer process (Chrome 85);
Electron web worker (Chrome 85).

Daninet commented 4 years ago

As far as I see there is a memory leak at the setInterval() call: it prevents GC from cleaning up the variables from the context. If I comment that out or if I add this line clearInterval(this.interval); to the .on('end') handler, I don't see those out of memory errors anymore.

In my experience, the average use case doesn't need web workers or worker threads. Usually the bottleneck is the disk I/O: the average consumer SSD cannot keep up with the rate of calculating hashes.

On their own, most algorithms from hash-wasm (including xxhash) shouldn't use more RAM than 1-2 MB / instance. Most browsers allow up to 2 / 4 GB of RAM allocated to WASM processes. So it shouldn't be a matter of concern if the code doesn't have any memory leaks.

aleksey-hoffman commented 4 years ago

@Daninet thank you for figuring out what was causing the problem. By the way, are you planning to add XXH3? It seems like it's 2x times faster than XXH64, according to the benchmark here: https://github.com/Cyan4973/xxHash

Daninet commented 4 years ago

I'm planning to add new algorithms, but I don't think that XXH3 would be significantly faster than XXH64 when running it in WASM. The performance of that algorithm comes from SSE2, which I cannot use from WASM, until the SIMD instructions aren't supported by the browsers.

aleksey-hoffman commented 4 years ago

@Daninet ahh, I see, you're compiling WASM directly into JS so it can be used in browsers as well. I'm personally using it in an Electron project.

Have you ever considered compiling your WASM module into C++ so it can be used as a native module in Node.js and Electron projects via N-API? Would that allow you to use SSE instructions?

Daninet commented 4 years ago

The main goal of this library is to provide a portable but fast solution for calculating hashes. The main target is the browser.

There are other of libraries on NPM, which use N-API, NAN or other ways to generate native bindings for Node.js / Electron. N-API allows using all types of custom instructions, but it is not portable: the source needs to be recompiled for each platform / CPU architecture where you want to run it.

jayphelps commented 4 years ago

Side note in case it wasn't known: support for SIMD in wasm V8 (chrome/node) can be enabled with --experimental-wasm-simd. Since this project is using emscripten: https://emscripten.org/docs/porting/simd.html

Daninet commented 4 years ago

Yeah, I know. The SIMD instructions are in origin trial currently. This means that probably in a few months Chrome will enable it by default. https://www.chromestatus.com/feature/6533147810332672 The main question is about Safari, which usually lags behind in terms of supporting new features.

I also made some measurements and with the current V8 version, I see about 10% improvement when using SIMD. Probably that will improve as the JIT gets better at optimizing SIMD instructions, but we will have to wait for that. I was thinking about compiling two versions from each library, but I think it's not worth to double the bundle size for that 10% improvement, which cannot be used anywhere without enabling flags. I also started writing a library to backport the SIMD instructions in the browser on runtime, where they aren't supported: https://github.com/Daninet/wasm-backporter It works, but it makes the performance worse, where the SIMD instructions are not supported.

I'm keeping an eye on the advancements regarding on SIMD. I will enable it as soon as it delivers significantly better performance and it has a decent browser support.

Daninet / hash-wasm