Fails to decode on large files (500 kb)

[wanderer]

Encoding works fine. But decode fails with

Error: Failed to parse
    at Decoder._decode (/home/null/code/js-primea-iframe-container/node_modules/borc/src/decoder.js:565:13)
    at Decoder.decodeFirst (/home/null/code/js-primea-iframe-container/node_modules/borc/src/decoder.js:576:10)
    at Object.decode (/home/null/code/js-primea-iframe-container/node_modules/borc/src/decoder.js:600:16)
    at Object.<anonymous> (/home/null/code/js-primea-iframe-container/examples/ipfstest.js:6:6)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)

To repodece

const cbor = require('borc')
const fs = require('fs')

const content = fs.readFileSync(__dirname + '/rootContainer.bundle.js').toString()
const encoded = cbor.encode(content)

cbor.decode(encoded)

Where rootContainer.bundle.js is attached rootContainer.bundle.js.zip

[dignifiedquire]

You need to set the heap size for large values for decoding manually. The default value is 65536. See https://github.com/dignifiedquire/borc/blob/master/src/decoder.js#L21 for details

// manually setting the heap size
const dec = new Decoder({size: heapSize})
const res = dec.decodeFirst(mylargebuffer)

[wanderer]

@dignifiedquire still fails

/home/null/code/borc/src/decoder.js:565
      throw new Error('Failed to parse')
      ^

Error: Failed to parse
    at Decoder._decode (/home/null/code/borc/src/decoder.js:565:13)
    at Decoder.decodeFirst (/home/null/code/borc/src/decoder.js:576:10)
    at Object.<anonymous> (/home/null/code/js-primea-iframe-container/examples/ipfstest.js:8:5)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)
    at startup (bootstrap_node.js:158:16)

[wanderer]

maybe its being encoded wrong

[wanderer]

nope, the encoding looks to be correct accouding to http://cbor.me/?bytes= . The error is buried some in decoder.asm.js. Is the c souce for that?

[dignifiedquire]

that's written by hand, no source

[wanderer]

ok so i noticed even though i have const dec = new cbor.Decoder({size: 100000000}) the decode always fail with any buffer bigger then 65536 bytes

[JohnRSim]

@wanderer did you make any progress or swap from CBOR to something else? I'm having the same issue 300kb file

[dignifiedquire]

Okay I checked out the source and it turns out the reason is that decoding of byte_string_32 and byte_string_64 are simply not implemented yet.

I pushed a branch with some failing test cases and a start of working on this, but fixing this will require a good deal of reshuffeling in decoder.asm.js as the current offset variable is assumed to be 32bit, but with these cases it is larger than that so I will need to introduce a bigint type into there..

[dignifiedquire]

ref https://github.com/dignifiedquire/borc/tree/fix-large

[dignifiedquire]

Turns out for 32bit sized ones I could do with the same structure. Fix is here and can handle now the sizes you were mentioning

[dignifiedquire]

@JohnRSim @wanderer would be great if you could checkout https://github.com/dignifiedquire/borc/pull/24 and confirm if this works for you

[JohnRSim]

After playing around - it started all working for me - and I was no longer getting the issue. I was using this to load the file in

fs.readFileSync(jsonFile, 'utf8');