ethersphere / bee-js

Javascript client library for connecting to Bee decentralised storage
https://bee-js.ethswarm.org/
BSD 3-Clause "New" or "Revised" License
60 stars 27 forks source link

RangeError: Too many properties to enumerate when uploading #971

Open Cafe137 opened 2 weeks ago

Cafe137 commented 2 weeks ago
RangeError: Too many properties to enumerate
    at Function.keys (<anonymous>)
    at Object.isEmptyObject (/Users/aron/Code/Swarm/swarmwiki/node_modules/axios/lib/utils.js:134:24)
    at getMergedValue (/Users/aron/Code/Swarm/swarmwiki/node_modules/axios/lib/core/mergeConfig.js:21:22)
    at valueFromConfig2 (/Users/aron/Code/Swarm/swarmwiki/node_modules/axios/lib/core/mergeConfig.js:43:14)
    at computeConfigValue (/Users/aron/Code/Swarm/swarmwiki/node_modules/axios/lib/core/mergeConfig.js:98:23)
    at Object.forEach (/Users/aron/Code/Swarm/swarmwiki/node_modules/axios/lib/utils.js:281:10)
    at mergeConfig (/Users/aron/Code/Swarm/swarmwiki/node_modules/axios/lib/core/mergeConfig.js:96:9)
    at Axios.request (/Users/aron/Code/Swarm/swarmwiki/node_modules/axios/lib/core/Axios.js:41:12)
    at wrap (/Users/aron/Code/Swarm/swarmwiki/node_modules/axios/lib/helpers/bind.js:5:15)
    at http (file:///Users/aron/Code/Swarm/swarmwiki/node_modules/@ethersphere/bee-js/dist/mjs/utils/http.js:23:28)

The file that triggers this:

X/fulltext/xapian 356114432 application/octet-stream+xapian

When Bee-JS code transforms the request configuration (including body) before handing it over to the HTTP client (axios), I suspect that the body is not detected as data but as a JS object instead, and it wrongly prepares it by extracting all of its indices (1 byte each if it is a Buffer/ArrayBuffer/Uint8Array/etc.) to the merged object.

Edit: see the comment below, the problem may not be in bee-js.

I see this error rarely, and IIRC it is always binary large files (threshold somewhere around 100 or 300MB).

Cafe137 commented 2 weeks ago

(Probably) a different variation of this is the following:

RangeError: Invalid array length
    at Function.keys (<anonymous>)
    at Object.isEmptyObject (/Users/aron/Code/Swarm/sandbox/node_modules/axios/lib/utils.js:134:24)
    at getMergedValue (/Users/aron/Code/Swarm/sandbox/node_modules/axios/lib/core/mergeConfig.js:21:22)
    at valueFromConfig2 (/Users/aron/Code/Swarm/sandbox/node_modules/axios/lib/core/mergeConfig.js:43:14)
    at computeConfigValue (/Users/aron/Code/Swarm/sandbox/node_modules/axios/lib/core/mergeConfig.js:98:23)
    at Object.forEach (/Users/aron/Code/Swarm/sandbox/node_modules/axios/lib/utils.js:281:10)
    at mergeConfig (/Users/aron/Code/Swarm/sandbox/node_modules/axios/lib/core/mergeConfig.js:96:9)
    at Axios.request (/Users/aron/Code/Swarm/sandbox/node_modules/axios/lib/core/Axios.js:41:12)
    at wrap (/Users/aron/Code/Swarm/sandbox/node_modules/axios/lib/helpers/bind.js:5:15)
    at http (/Users/aron/Code/Swarm/sandbox/node_modules/@ethersphere/bee-js/dist/cjs/utils/http.js:32:52)

This is the axios function that triggers it:

/**
 * Determine if a value is a empty Object
 *
 * @param {Object} val The value to test
 * @return {boolean} True if value is a empty Object, otherwise false
 */
function isEmptyObject(val) {
  return val && Object.keys(val).length === 0 && Object.getPrototypeOf(val) === Object.prototype;
}

This is val that gets passed:

<Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 149999950 more bytes>

This seems to be an axios bug, in which it tries to determine if a buffer is an object, and calling Object.keys on it kills the JS runtime