search

libp2p / js-libp2p

The JavaScript Implementation of libp2p networking stack.
https://libp2p.io
Other
2.3k stars 440 forks source link

`TypeError: TextDecoder is not a constructor` when handling multiaddresses in browser #909

Closed da-kami closed 3 years ago

da-kami commented 3 years ago

Type:

Bug

Severity:

Critical

Description:

When handling any kind of code that depends on https://github.com/multiformats/js-multibase the application crashes in the Browser with:

TypeError: TextDecoder is not a constructor
./node_modules/multiaddr/node_modules/multibase/src/util.js
node_modules/multiaddr/node_modules/multibase/src/util.js:6

  3 | // @ts-ignore
  4 | const { TextEncoder, TextDecoder } = require('web-encoding')
  5 | 
> 6 | const textDecoder = new TextDecoder()
  7 | /**
  8 |  * @param {ArrayBufferView|ArrayBuffer} bytes
  9 |  * @returns {string}

Test with Firefox and Chrome.

Using require for importing resulted in the same error. Using multiaddr function like shown in the chat example https://github.com/libp2p/js-libp2p/blob/master/examples/chat/src/dialer.js#L33 resulted in the same error.

The problem seems to be related to how imports are handled in multibase.

Steps to reproduce the error:

I ran into this problem when building a PoC trying to connect to a rust-libp2p daemon with js-libp2p from the browser: https://github.com/da-kami/js-libp2p-quote-poc

Note: I also did a pure Javascript version which failed with the same error.

Are we missing something for working with Mutiaddress in the browser or is this a bug?

vasco-santos commented 3 years ago

Hello @da-kami We are working on updating all the dependencies to not use web-encoding, since we really not need it #904 . Meanwhile, for this version we are blocked on https://github.com/Gozala/web-encoding/pull/21

da-kami commented 3 years ago

Hello @da-kami We are working on updating all the dependencies to not use web-encoding, since we really not need it #904 . Meanwhile, for this version we are blocked on Gozala/web-encoding#21

Thanks for the quick reply! I subscribed to #904 but after a quick look on the dependencies, I am not sure #904 will resolve not depending on web-encoding anymore:

As far as I could see #904 updates multiaddr to 9.0.1 which depends on multibase 0.4.2, but that still depends on web-encoding:

https://github.com/multiformats/js-multiaddr/blob/v9.0.1/package.json#L41 https://github.com/multiformats/js-multibase/blob/v4.0.2/package.json#L35

da-kami commented 3 years ago

As a temporary quickfix we use yarn's resolutions feature to upgrade multibase and uint8arrays to the latest version which removes web-encoding:

https://github.com/da-kami/js-libp2p-quote-poc/blob/master/package.json#L23-L26

vasco-santos commented 3 years ago

Thanks for the quick reply! I subscribed to #904 but after a quick look on the dependencies, I am not sure #904 will resolve not depending on web-encoding anymore:

As far as I could see #904 updates multiaddr to 9.0.1 which depends on multibase 0.4.2, but that still depends on web-encoding:

multibase@4.0.3 was already shipped removing web-encoding. You can already install libp2p@0.31.0-rc.0 and try it out

da-kami commented 3 years ago

Thanks for the quick reply! I subscribed to #904 but after a quick look on the dependencies, I am not sure #904 will resolve not depending on web-encoding anymore:

As far as I could see #904 updates multiaddr to 9.0.1 which depends on multibase 0.4.2, but that still depends on web-encoding:

multibase@4.0.3 was already shipped removing web-encoding. You can already install libp2p@0.31.0-rc.0 and try it out

Thanks! I gave 0.31.0-rc.0 it a try earlier and ran into some other issues. Will see to report them in detail once I find time to dig through.

D4nte commented 3 years ago

Yes, facing an issue too when trying the release candidate:

../node_modules/libp2p/src/upgrader.js 315:14
Module parse failed: Cannot use keyword 'await' outside an async function (315:14)
File was processed with these loaders:
 * ./node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
|             stream,
|             protocol
>           } = await mss.select(protocols);
|           if (this.metrics) this.metrics.trackStream({
|             stream,
vasco-santos commented 3 years ago

The reported error is misleading as that await is wrapped by an async function: https://github.com/libp2p/js-libp2p/blob/master/src/upgrader.js#L257

I just tried our libp2p-in-the-browser example, which uses parcel+babel and does not have this issue. Per the original post, I see that you are using CRA for your project and this might be related to how dependencies are bundled.

We are checking the RC with js-ipfs that has an example using CRA as well, I will keep you posted if this leads to the same issue

vasco-santos commented 3 years ago

@D4nte we found the bug for CRA. You can try the latest rc: 0.31.0-rc.3.

This was the fix. Somehow CRA with inline types comments understands an async function as a regular function 🤷🏼

FYI, we will be shipping the final release soon

D4nte commented 3 years ago

@D4nte we found the bug for CRA. You can try the latest rc: 0.31.0-rc.3.

This was the fix. Somehow CRA with inline types comments understands an async function as a regular function 🤷🏼

FYI, we will be shipping the final release soon

@vasco-santos great thanks! I actually quickly tried rc.3 yesterday and it looked good but I had to log off so did not have time to double check. Will report once I do more testing.

D4nte commented 3 years ago

Ok I can confirm the error is gone with rc.3 :)

D4nte commented 3 years ago

I am now getting ReferenceError: TextDecoder is not defined when:

Is that a known issue?

vasco-santos commented 3 years ago

Can you let me know your libp2p dependencies versions? Also do npm ls multibase

D4nte commented 3 years ago

In the react app:

web-chat@0.1.0 /home/froyer/src/status-im/js-waku/web-chat
└─┬ peer-id@0.14.8
  ├─┬ cids@1.1.6
  │ └── multibase@4.0.4
  ├─┬ libp2p-crypto@0.19.4
  │ └── multibase@4.0.4 deduped
  ├─┬ multihashes@4.0.2
  │ └── multibase@4.0.4 deduped
  └─┬ uint8arrays@2.1.5
    └── multibase@4.0.4 deduped

In the js-waku library that is used by the react app and brings in libp2p:

js-waku@1.0.0 /home/froyer/src/status-im/js-waku
├─┬ libp2p-gossipsub@0.7.0
│ ├─┬ libp2p-interfaces@0.7.2
│ │ ├─┬ libp2p-crypto@0.18.0
│ │ │ └── multibase@3.1.2 deduped
│ │ ├─┬ multiaddr@8.1.2
│ │ │ └── multibase@3.1.2 deduped
│ │ ├── multibase@3.1.2
│ │ └─┬ multihashes@3.1.2
│ │   ├── multibase@3.1.2 deduped
│ │   └─┬ uint8arrays@2.1.3
│ │     └── multibase@4.0.2
│ ├─┬ peer-id@0.14.3
│ │ ├─┬ multihashes@3.1.2
│ │ │ └── multibase@3.1.2 deduped
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2
│ └─┬ uint8arrays@1.1.0
│   └── multibase@3.1.2 deduped
├─┬ libp2p-noise@2.0.5
│ ├─┬ libp2p-crypto@0.19.0
│ │ └── multibase@3.1.2 deduped
│ └─┬ uint8arrays@2.1.3
│   └── multibase@4.0.2
├─┬ libp2p-secio@0.13.1
│ ├─┬ libp2p-crypto@0.18.0
│ │ └── multibase@3.1.2 deduped
│ ├─┬ multiaddr@8.1.2
│ │ └── multibase@3.1.2 deduped
│ └─┬ multihashing-async@2.1.2
│   ├─┬ multihashes@4.0.2
│   │ ├── multibase@4.0.2
│   │ └─┬ uint8arrays@2.1.3
│   │   └── multibase@4.0.2 deduped
│   └─┬ uint8arrays@2.1.3
│     └── multibase@4.0.2
├─┬ libp2p-tcp@0.15.3
│ ├─┬ libp2p-utils@0.2.3
│ │ └─┬ multiaddr@8.1.2
│ │   └── multibase@3.1.2 deduped
│ ├─┬ mafmt@8.0.4
│ │ └─┬ multiaddr@8.1.2
│ │   └── multibase@3.1.2 deduped
│ └─┬ multiaddr@8.1.2
│   └── multibase@3.1.2 deduped
├─┬ libp2p-websockets@0.15.5
│ └─┬ multiaddr-to-uri@6.0.0
│   └─┬ multiaddr@8.1.2
│     └── multibase@3.1.2 deduped
├─┬ libp2p@0.31.0-rc.3
│ ├─┬ cids@1.1.6
│ │ ├── multibase@4.0.2
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2 deduped
│ ├─┬ libp2p-interfaces@0.10.1
│ │ ├── multibase@4.0.4
│ │ └─┬ uint8arrays@2.1.5
│ │   └── multibase@4.0.4 deduped
│ ├─┬ multicodec@3.0.1
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2
│ └─┬ multistream-select@2.0.0
│   └─┬ uint8arrays@2.1.5
│     └── multibase@4.0.4
└─┬ multiaddr@9.0.1
  ├── multibase@4.0.4
vasco-santos commented 3 years ago

This error comes from outdated multibase. You can already update everything in there except gossipsub (still needs https://github.com/ChainSafe/js-libp2p-gossipsub/pull/153 merged)

You can see on https://github.com/libp2p/js-libp2p/pull/912 what modules versions you should have. You can update everything except gossip, including new libp2p rc and you can experiment with gossipsub branch referenced for full functionality.

Sorry for the delay, we are working on guaranteeing everything is working properly with js-ipfs before shipping a final release

D4nte commented 3 years ago

This error comes from outdated multibase. You can already update everything in there except gossipsub (still needs ChainSafe/js-libp2p-gossipsub#153 merged)

You can see on #912 what modules versions you should have. You can update everything except gossip, including new libp2p rc and you can experiment with gossipsub branch referenced for full functionality.

Looks like gossipsub is released too. However, it seems that peer-id is still bringing an old multibase version:

▶ npm ls multibase   
web-chat@0.1.0 /home/froyer/src/status-im/js-waku/web-chat
└─┬ peer-id@0.14.8
  ├─┬ cids@1.1.6
  │ └── multibase@4.0.4
  ├─┬ libp2p-crypto@0.19.4
  │ └── multibase@4.0.4 deduped
  ├─┬ multihashes@4.0.2
  │ └── multibase@4.0.4 deduped
  └─┬ uint8arrays@2.1.5
    └── multibase@4.0.4 deduped

Sorry for the delay, we are working on guaranteeing everything is working properly with js-ipfs before shipping a final release

No worries, thanks for the quick turn around!

D4nte commented 3 years ago

Wait, which multibase version is the outdated one?

▶ npm ls multibase
js-waku@1.0.0 /home/froyer/src/status-im/js-waku
├─┬ libp2p-gossipsub@0.9.0
│ ├─┬ libp2p-interfaces@0.10.3
│ │ ├── multibase@4.0.4
│ │ ├─┬ multihashes@4.0.2
│ │ │ ├── multibase@4.0.2
│ │ │ └─┬ uint8arrays@2.1.3
│ │ │   └── multibase@4.0.2 deduped
│ │ └─┬ uint8arrays@2.1.5
│ │   └── multibase@4.0.4 deduped
│ ├─┬ peer-id@0.14.3
│ │ ├─┬ multihashes@3.1.2
│ │ │ └── multibase@3.1.2 deduped
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2
│ └─┬ uint8arrays@2.1.5
│   └── multibase@4.0.4
├─┬ libp2p-noise@3.0.0
│ ├─┬ libp2p-crypto@0.19.0
│ │ └── multibase@3.1.2
│ └─┬ uint8arrays@2.1.3
│   └── multibase@4.0.2
├─┬ libp2p-secio@0.13.1
│ ├─┬ libp2p-crypto@0.18.0
│ │ └── multibase@3.1.2 deduped
│ ├─┬ multiaddr@8.1.2
│ │ └── multibase@3.1.2 deduped
│ ├─┬ multihashing-async@2.1.2
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2
│ └─┬ uint8arrays@1.1.0
│   └── multibase@3.1.2 deduped
├─┬ libp2p@0.31.0
│ ├─┬ cids@1.1.6
│ │ ├── multibase@4.0.2
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2 deduped
│ ├─┬ interface-datastore@4.0.0
│ │ └─┬ uint8arrays@2.1.5
│ │   └── multibase@4.0.4
│ ├─┬ multicodec@3.0.1
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2
│ └─┬ multistream-select@2.0.0
│   └─┬ uint8arrays@2.1.5
│     └── multibase@4.0.4
└─┬ multiaddr@9.0.1
  ├── multibase@4.0.4
  └─┬ uint8arrays@2.1.5
    └── multibase@4.0.4 deduped
D4nte commented 3 years ago

I having the issue with multibase 4.0.4, the latest version. This is not expected, right?

 FAIL  src/command.test.ts
  ● Test suite failed to run

    ReferenceError: TextDecoder is not defined

    > 1 | import { multiaddr } from 'multiaddr';
        | ^
      2 | import PeerId from 'peer-id';
      3 | import Waku from '../../build/main/lib/waku';
      4 |

      at Object.<anonymous> (../node_modules/multibase/src/util.js:3:21) <-- This is multibase 4.0.4
      at Object.<anonymous> (../node_modules/multibase/src/base.js:3:24)
      at Object.<anonymous> (../node_modules/multibase/src/constants.js:4:14)
      at Object.<anonymous> (../node_modules/multibase/src/index.js:7:19)
      at Object.<anonymous> (../node_modules/uint8arrays/to-string.js:3:32)
      at Object.<anonymous> (../node_modules/multiaddr/src/ip.js:4:28)
      at Object.<anonymous> (../node_modules/multiaddr/src/convert.js:3:12)
      at Object.<anonymous> (../node_modules/multiaddr/src/codec.js:3:17)
      at Object.<anonymous> (../node_modules/multiaddr/src/index.js:3:15)
      at Object.<anonymous> (src/command.ts:1:1)
      at Object.<anonymous> (src/command.test.ts:1:1)
vasco-santos commented 3 years ago

Wait, which multibase version is the outdated one?

everything in multibase should be 4+

I having the issue with multibase 4.0.4, the latest version. This is not expected, right?

The error now looks different then before. What node version are you using? You should be using 14 or 15 by now?

A couple of other questions per latest dependency tree:

I think you need to reinstall the dependencies without your package lock, you have several versions that are not in the latest ones

If you cannot solve this issue with the latest issues, please let me know if I can try to replicate this locally?

D4nte commented 3 years ago

Thanks. So there are still some multibase 3 coming from libp2p-noise for example:

▶ npm ls multibase
js-waku@1.0.0 /home/froyer/src/status-im/js-waku
├─┬ libp2p-gossipsub@0.9.0
│ ├─┬ libp2p-interfaces@0.10.3
│ │ ├── multibase@4.0.4
│ │ ├─┬ multihashes@4.0.2
│ │ │ ├── multibase@4.0.2
│ │ │ └─┬ uint8arrays@2.1.3
│ │ │   └── multibase@4.0.2 deduped
│ │ └─┬ uint8arrays@2.1.5
│ │   └── multibase@4.0.4 deduped
│ ├─┬ peer-id@0.14.3
│ │ ├─┬ multihashes@3.1.2
│ │ │ └── multibase@3.1.2 deduped
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2
│ └─┬ uint8arrays@2.1.5
│   └── multibase@4.0.4
├─┬ libp2p-noise@3.0.0
│ ├─┬ libp2p-crypto@0.19.0
│ │ ├── multibase@3.1.2
│ │ └─┬ uint8arrays@1.1.0
│ │   └── multibase@3.1.2 deduped
│ └─┬ uint8arrays@2.1.3
│   └── multibase@4.0.2
├─┬ libp2p@0.31.0
│ ├─┬ cids@1.1.6
│ │ ├── multibase@4.0.2
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2 deduped
│ ├─┬ interface-datastore@4.0.0
│ │ └─┬ uint8arrays@2.1.5
│ │   └── multibase@4.0.4
│ ├─┬ multicodec@3.0.1
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2
│ ├─┬ multihashing-async@2.1.2
│ │ └─┬ uint8arrays@2.1.3
│ │   └── multibase@4.0.2
│ └─┬ multistream-select@2.0.0
│   └─┬ uint8arrays@2.1.5
│     └── multibase@4.0.4
└─┬ multiaddr@9.0.1
  ├── multibase@4.0.4
  └─┬ uint8arrays@2.1.5
    └── multibase@4.0.4 deduped

The error now looks different then before. What node version are you using? You should be using 14 or 15 by now?

The errors come from using react-scripts@4.0.3

  • any reason to use secio? We deprecated it one year ago and it should not be used.

Indeed, it was an unused dependency, now removed. Thanks!

  • why do you have the PeerId version fixed? It should be in the latest as well

Indeed, now removed.

vasco-santos commented 3 years ago

You need to have some kind of lock file @D4nte

When you install libp2p-noise for example, it will install the latest of the 0.19.x series of libp2p-crypto, which will be 0.19.4 and will come with multibase 4. Same for gossipsub and its PeerId

D4nte commented 3 years ago

You need to have some kind of lock file @D4nte

When you install libp2p-noise for example, it will install the latest of the 0.19.x series of libp2p-crypto, which will be 0.19.4 and will come with multibase 4. Same for gossipsub and its PeerId

Yes I have a package-lock.json file but thank you for the tip, I'll play around with that!

vasco-santos commented 3 years ago

This should be fixed with the new libp2p release and all the latest libp2p modules. Feel free to re-open if this is still a problem for you

D4nte commented 3 years ago

I am having the related issue again:

  ● Test suite failed to run

    ReferenceError: TextDecoder is not defined

    > 1 | import Libp2p, { Connection, Libp2pModules, Libp2pOptions } from 'libp2p';
        | ^
      2 | import Mplex from 'libp2p-mplex';
      3 | import { bytes } from 'libp2p-noise/dist/src/@types/basic';
      4 | import { Noise } from 'libp2p-noise/dist/src/noise';

      at Object.<anonymous> (../../node_modules/multibase/src/util.js:3:21)
      at Object.<anonymous> (../../node_modules/multibase/src/base.js:3:24)
      at Object.<anonymous> (../../node_modules/multibase/src/constants.js:4:14)
      at Object.<anonymous> (../../node_modules/multibase/src/index.js:7:19)
      at Object.<anonymous> (../../node_modules/multihashes/src/index.js:6:19)
      at Object.<anonymous> (../../node_modules/peer-id/src/index.js:7:12)
      at Object.<anonymous> (../../node_modules/libp2p/src/index.js:10:16)
      at Object.<anonymous> (../../src/lib/waku.ts:1:1)
      at Object.<anonymous> (../../src/index.ts:3:1)
      at Object.<anonymous> (src/App.tsx:5:1)
      at Object.<anonymous> (src/App.test.tsx:3:1)

When running npm run test:unit from an example folder examples/eth-dm. The eth-dm project does not have multibase installed:

js-waku/examples/eth-dm  eth-dm ✔                                                           17h35m  
▶ npm ls multibase
eth-dm@0.1.0 /home/froyer/src/status-im/js-waku/examples/eth-dm
└── (empty)

And the parent project has latest multibase only:

src/status-im/js-waku  eth-dm ✔                                                             17h36m  
▶ npm ls multibase
js-waku@0.7.0 /home/froyer/src/status-im/js-waku
├─┬ libp2p-gossipsub@0.10.0
│ ├─┬ libp2p-interfaces@0.11.0
│ │ ├── multibase@4.0.4 deduped
│ │ └─┬ multihashes@4.0.2
│ │   └── multibase@4.0.4 deduped
│ └─┬ uint8arrays@2.1.5
│   └── multibase@4.0.4 deduped
├─┬ libp2p-noise@3.0.0
│ └─┬ libp2p-crypto@0.19.4
│   └── multibase@4.0.4 deduped
├─┬ libp2p@0.31.7
│ ├─┬ cids@1.1.7
│ │ └── multibase@4.0.4 deduped
│ └─┬ libp2p-interfaces@0.10.4
│   └── multibase@4.0.4 deduped
└─┬ multiaddr@9.0.1
  └── multibase@4.0.4

and the multibase line that returns the error is indeed 4.0.4:

js-waku/examples/eth-dm  eth-dm ✔                                                           17h37m  
▶ cat ../../node_modules/multibase/package.json | jq .version
"4.0.4"

The code can be found at https://github.com/status-im/js-waku/pull/212

What am I missing?

vasco-santos commented 3 years ago

Is this still a problem? The PR was merged

From an initial observation, the error seems different, probably using the latest https://github.com/multiformats/js-multibase/blob/master/src/util.js#L3 . So what environment is this running?

D4nte commented 3 years ago

It's running in react-scripts which I assume is the source of the problem. Apparently this error can appear when using older node versions but I use node 16.