build error: First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object

[lemoustachiste]

Hi,

I have a project with an older version of cbor (5.0.1). First off, I have tried to update to the latest version (7.0.5) but that didn't fix the issue, and yields another issue at build time, with rollup, so I'd rather not go down that path just yet.

The issue occurs in a jest environment, when testing the build code. That is, if I run a test against the source code, there is no issue and I get the correct output.

If I test against the build code, then this error arises:

First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object

with the stack call as follows

fromObject (dist/decoder-es.js:480:9)
      at from (dist/decoder-es.js:314:10)
      at new subarray (dist/decoder-es.js:290:10)
          at Uint8Array.subarray (<anonymous>)
      at Uint8Array.slice (dist/decoder-es.js:1249:19)
      at fromListPartial (dist/decoder-es.js:4209:26)
      at fromList (dist/decoder-es.js:4196:11)
      at NoFilter.Object.<anonymous>.Readable.read (dist/decoder-es.js:3746:20)
      at NoFilter.read (dist/decoder-es.js:8626:14)
      at Object.decode [as decodeFirstSync] (dist/decoder-es.js:10726:19)
      at decode (dist/decoder-es.js:30392:22)
      at Object.<anonymous> (test/build.es.test.js:7:24)

I have set up a repro here if that helps: https://github.com/lemoustachiste/repro-cbor-build. It seems to me that it was until recently working (albeit not in this repo), but I am not capable to identify what changed so maybe you can help me get on the right tracks.

The entry point is here: https://github.com/lemoustachiste/repro-cbor-build/blob/master/src/index.js#L97

Thanks

[hildjj]

I think this is a problem with the Readable shim that's getting installed. Is the build issue you're getting on 7.0.5 on node-inspect-extracted? I'm looking into that too.

[hildjj]

node-inspect-extracted 1.0.7 should fix the build issue with cbor 7.0.5. You'll also either need to take a dependency on bignumber.js or mark it as external. I'm now going to look at your actual issue. :)

[hildjj]

When I switched to async mode, I got a lot farther:

export default async function decode (base58string) {
    const encoded = multibase.decode(base58string)
    // decode -> await decodeFirst
    const map = await cbor.decodeFirst(encoded)
    const json = constructRootJSON(map)

    return json
}

Now your tests fail for substantive reasons. I bet the sync read() functionality in the ReadStream shim isn't quite the same as node's. How important is synchronous behavior to you?

[lemoustachiste]

@hildjj thanks for looking at my issue.

On your first point, correct, I was getting the error with node-inspect-extracted and it is now resolved so I am able to build. Thanks for that.

On the second point, moving to async didn't change the error which remains the same. Could you share what the output of the test was? To me async is fine, if I can get it to work :)

[lemoustachiste]

ok I have updated some rollup dependencies on the repro (and should on my project too), the error is a little bit less cryptic now

Expected  -   2
    + Received  + 142

      Object {
        "anchors": Array [
          "blink:btc:testnet:1d0005700ece4a522bcf6e3ec1dc2821c1c304332b93d6148d8cec47f21ab89a",
        ],
    -   "merkleRoot": "2c7afa4f8192bd8d0e243da2044306b2183527270ef6fd76854c34a1288756ba",
    +   "merkleRoot": Object {
    +     "data": Array [
    +       120,
    +       64,
    +       50,
    +       99,
    +       55,
    +       97,
    +       102,
    +       97,
    +       52,
    +       102,
    +       56,
    +       49,
    +       57,
    +       50,
    +       98,
    +       100,
    +       56,
    +       100,
    +       48,
    +       101,
    +       50,
    +       52,
    +       51,
    +       100,
    +       97,
    +       50,
    +       48,
    +       52,
    +       52,
    +       51,
    +       48,
    +       54,
    +       98,
    +       50,
    +       49,
    +       56,
    +       51,
    +       53,
    +       50,
    +       55,
    +       50,
    +       55,
    +       48,
    +       101,
    +       102,
    +       54,
    +       102,
    +       100,
    +       55,
    +       54,
    +       56,
    +       53,
    +       52,
    +       99,
    +       51,
    +       52,
    +       97,
    +       49,
    +       50,
    +       56,
    +       56,
    +       55,
    +       53,
    +       54,
    +       98,
    +       97,
    +     ],
    +     "type": "Buffer",
    +   },
        "path": Array [
          Object {
            "left": "e1b59fcf59c7a11d725935dd72520268ca715b754accccb896a0cc1f765056db",
          },
    +   ],
    +   "targetHash": Object {
    +     "data": Array [
    +       120,
    +       64,
    +       53,
    +       99,
    +       49,
    +       102,
    +       98,
    +       101,
    +       100,
    +       54,
    +       100,
    +       55,
    +       100,
    +       49,
    +       98,
    +       99,
    +       50,
    +       54,
    +       53,
    +       50,
    +       99,
    +       57,
    +       52,
    +       97,
    +       51,
    +       49,
    +       57,
    +       49,
    +       52,
    +       48,
    +       101,
    +       52,
    +       50,
    +       99,
    +       52,
    +       55,
    +       97,
    +       49,
    +       53,
    +       52,
    +       99,
    +       100,
    +       55,
    +       55,
    +       48,
    +       97,
    +       101,
    +       50,
    +       48,
    +       99,
    +       55,
    +       101,
    +       102,
    +       49,
    +       54,
    +       98,
    +       98,
    +       53,
    +       57,
    +       98,
    +       99,
    +       54,
    +       54,
    +       53,
    +       52,
    +       100,
          ],
    -   "targetHash": "5c1fbed6d7d1bc2652c94a319140e42c47a154cd770ae20c7ef16bb59bc6654d",
    +     "type": "Buffer",
    +   },

So it does not seem to reconcile the Buffer to an actual string. I'll keep digging.

[hildjj]

We're getting the same results now. I'll also take another look.

[lemoustachiste]

I have found that it's when updating to @rollup/plugin-node-resolve that it changes the output, if that helps.

[hildjj]

Can you push your changes to lemoustachiste/repro-cbor-build, please, so we're working from the same starting point?

[lemoustachiste]

@hildjj done. I have tested with node 13 and node 15, same issue with both. I have removed the async call since it does not change anything either to the problem.

[hildjj]

OK, i've got it. Use decode(encoded, {preferWeb: true}), then in the places where you are looking for Buffer, look for Uint8Array. The environment that the babel'd code is running in is using slightly different Buffer implementations, but the native Uint8Arrays will make it through all of that cleanly.

        if (value instanceof Uint8Array) {
            value = cbor.decode(value).toString('hex')
        }

        acc[key] = value
        return acc
    }, {})
}

export default function decode (base58string) {
    const encoded = multibase.decode(base58string)
    const map = cbor.decode(encoded, {preferWeb: true})
    const json = constructRootJSON(map)

    return json
}

[lemoustachiste]

Thanks @hildjj, that fixes it in the repro indeed. Let me try and check in the real life project, it might not be an immediate process as this issue comes from a third party. I'll keep you posted.

Thanks again for the quick support.

[hildjj]

Happy to help. I've had a ton of issues where one class is imported in two different ways, and breaks instanceof but most other things work. For instance, this happens with the vm module in node, which wraps proxies around everything.

[hildjj]

There's another (brittle) approach you can use if you can't update to a version of cbor that has preferWeb:

if (value.constructor.name === 'Buffer') {

This obviously isn't great, but it should work in most real-world scenarios.

[lemoustachiste]

I have isolated the issue to the type of the value property, so even preferWeb is not required. For all intent of purposes I'd say this issue is resolved so I am going to close.

Thanks again @hildjj