search

solana-labs / solana-web3.js

Solana JavaScript SDK
https://solana-labs.github.io/solana-web3.js
MIT License
2.14k stars 853 forks source link

web3.js: Remove use of Buffer #1100

Open macalinao opened 2 years ago

macalinao commented 2 years ago

Problem

Buffer isn't built for the web. Uint8Array is and saves 10kb gzipped.

Proposed Solution

Begin migrating code to use Uint8Array instead. Code that doesn't use Borsh should get optimized automatically.

steveluscher commented 2 years ago

100%. This is another change that we should negotiate with near/borsh-js to eliminate this dependency from the web bundle entirely, everywhere. cc/ @ailisp.

ailisp commented 2 years ago

Your treeshaking proposal is impressive! We also have plan to drop Buffer-usage in borsh. Uint8Array is available for all, <10-year browsers. The only challenge part is to borsh serialize a string, it uses Buffer.from(string, 'utf-8'). Standard web api for doing that is TextEncoder, but that need polyfills.

macalinao commented 2 years ago

Your treeshaking proposal is impressive! We also have plan to drop Buffer-usage in borsh. Uint8Array is available for all, <10-year browsers. The only challenge part is to borsh serialize a string, it uses Buffer.from(string, 'utf-8'). Standard web api for doing that is TextEncoder, but that need polyfills.

Yeah Uint8Array has good polyfills so I think it's fine-- virtually all frontend packaging tools have some form of browserslist/polyfilling in their build configs.

10kb gzipped if you guys accomplish this. And no more browserify! 🙂

katopz commented 1 year ago

Glad to hear this! node buffer shim took me almost 3 hours to make it to work for both dev and prod via vite. 🤷‍♂️

mindrunner commented 1 year ago

Is this already work in progress? I might want to have a look at this. Does anyone know how big/complex this migration gonna be?

steveluscher commented 1 year ago

My focus is on replacing web3.js altogether. Follow along in #1111 and in all of the new packages in the packages/ directory.

Deprecating Buffer in the current API would probably take just as long as rewriting the whole thing. Backward compat is the hardest thing, because Uint8Array and Buffer behave slightly differently (particularly toString()) and it’s impossible to know who depends on which of the peculiar behaviours.

val-samonte commented 6 months ago

Sorry for being that guy but any update on this? This prevents me to use vite before.

buffalojoec commented 6 months ago

Sorry for being that guy but any update on this? This prevents me to use vite before.

Unfortunately we can't save the legacy library from Buffer, but the new (2.0) release is going to be all Uint8Array. You can see preview release candidates here: https://github.com/solana-labs/solana-web3.js/releases