web3.js: Remove use of Buffer

[macalinao]

Problem

Buffer isn't built for the web. Uint8Array is and saves 10kb gzipped.

Proposed Solution

Begin migrating code to use Uint8Array instead. Code that doesn't use Borsh should get optimized automatically.

[steveluscher]

100%. This is another change that we should negotiate with near/borsh-js to eliminate this dependency from the web bundle entirely, everywhere. cc/ @ailisp.

[ailisp]

Your treeshaking proposal is impressive! We also have plan to drop Buffer-usage in borsh. Uint8Array is available for all, <10-year browsers. The only challenge part is to borsh serialize a string, it uses Buffer.from(string, 'utf-8'). Standard web api for doing that is TextEncoder, but that need polyfills.

[macalinao]

Your treeshaking proposal is impressive! We also have plan to drop Buffer-usage in borsh. Uint8Array is available for all, <10-year browsers. The only challenge part is to borsh serialize a string, it uses Buffer.from(string, 'utf-8'). Standard web api for doing that is TextEncoder, but that need polyfills.

Yeah Uint8Array has good polyfills so I think it's fine-- virtually all frontend packaging tools have some form of browserslist/polyfilling in their build configs.

10kb gzipped if you guys accomplish this. And no more browserify! 🙂

[katopz]

Glad to hear this! node buffer shim took me almost 3 hours to make it to work for both dev and prod via vite. 🤷‍♂️

[mindrunner]

Is this already work in progress? I might want to have a look at this. Does anyone know how big/complex this migration gonna be?

[steveluscher]

My focus is on replacing web3.js altogether. Follow along in #1111 and in all of the new packages in the packages/ directory.

Deprecating Buffer in the current API would probably take just as long as rewriting the whole thing. Backward compat is the hardest thing, because Uint8Array and Buffer behave slightly differently (particularly toString()) and it’s impossible to know who depends on which of the peculiar behaviours.

[val-samonte]

Sorry for being that guy but any update on this? This prevents me to use vite before.

[buffalojoec]

Sorry for being that guy but any update on this? This prevents me to use vite before.

Unfortunately we can't save the legacy library from Buffer, but the new (2.0) release is going to be all Uint8Array. You can see preview release candidates here: https://github.com/solana-labs/solana-web3.js/releases

[steveluscher]

@solana/web3.js from 2.0 onward is Buffer free.

[github-actions[bot]]

Because there has been no activity on this issue for 7 days since it was closed, it has been automatically locked. Please open a new issue if it requires a follow up.