search

dignifiedquire / pull-length-prefixed

Streaming length prefixed buffers with pull-streams
MIT License
8 stars 18 forks source link

Increasing number of source values causes "Maximum call stack size exceeded" in decode() #18

Open areth opened 5 years ago

areth commented 5 years ago

In this example stack explodes at about 500 values of source:

const pull = require('pull-stream')
const lp = require('pull-length-prefixed')
const assert = require('assert')

const datas = [...Array(500).keys()].map(() => Buffer.from('payload'))

pull(
  pull.values(datas),
  lp.encode(),
  pull.collect((err, encodedArray) => {
    if(err) {
      throw err
    }

    pull(
      pull.values(encodedArray),
      lp.decode(),
      pull.collect((err, array) => {
        if(err) {
          throw err
        }
        assert(JSON.stringify(array) === JSON.stringify(datas), 'Data corrupted')
        console.log('Done')
      })
    )
  })
)

This produces:

error uncaughtException: Maximum call stack size exceeded
RangeError: Maximum call stack size exceeded
    at _decodeFromReader (/lp-highload/node_modules/pull-length-prefixed/src/decode.js:26:29)
    at Object.reader.read [as cb] (/lp-highload/node_modules/pull-length-prefixed/src/decode.js:91:16)
    at drain (/lp-highload/node_modules/pull-reader/index.js:46:23)
    at /lp-highload/node_modules/pull-reader/index.js:63:18
    at /lp-highload/node_modules/pull-stream/sources/values.js:19:7
    at more (/lp-highload/node_modules/pull-reader/index.js:59:7)
    at Function.reader.read (/lp-highload/node_modules/pull-reader/index.js:99:7)
    at readByte (/lp-highload/node_modules/pull-length-prefixed/src/decode.js:89:12)
    at readVarintMessage (/lp-highload/node_modules/pull-length-prefixed/src/decode.js:85:32)
    at _decodeFromReader (/lp-highload/node_modules/pull-length-prefixed/src/decode.js:64:5)
    at next (/lp-highload/node_modules/pull-length-prefixed/src/decode.js:25:9)
    at _decodeFromReader (/lp-highload/node_modules/pull-length-prefixed/src/decode.js:29:11)
    at readMessage (/lp-highload/node_modules/pull-length-prefixed/src/decode.js:115:9)
    at Object.reader.read [as cb] (/lp-highload/node_modules/pull-length-prefixed/src/decode.js:127:5)
    at drain (/lp-highload/node_modules/pull-reader/index.js:39:14)
    at more (/lp-highload/node_modules/pull-reader/index.js:55:13)
    ...

Full log https://github.com/areth/lp-highload/blob/master/exceptions.log.bak Repo to reproduce https://github.com/areth/lp-highload

areth commented 5 years ago

The root of the issue in this recursion: https://github.com/dignifiedquire/pull-length-prefixed/blob/27d9f420c684ac7589dbac1b6986b68c7fadff0f/src/decode.js#L23-L30 To resolve the issue I was thinking about asynchronous excecution, and it works:

function next () {
  setImmediate(() => {
    _decodeFromReader(reader, opts, (err, msg) => {
      if (err) return p.end(err)

      p.push(msg)
      next()
    })
  })
}

But I strongly doubt that the implicit async is acceptable in the pull-stream ecosystem. It seems, that decode() have to be rewritten with a loop. I could do this. Please share your thoughts.