Compiled web3 bundle is HUGE

[josephg]

The browserify-compiled web3 bundle is crazy huge. I'm using the 1.0 beta and pre-minification web3 is 1.3 megs. Minification halves that, but its still way bigger than it needs to be.

$ cat foo.js 
require('web3')
$ browserify foo.js -o bundle.js
$ ls -l bundle.js 
-rw-r--r--  1 josephg  staff  1365873 15 Nov 16:00 bundle.js

Most of the bundle is transitive dependancies of dubious usefulness that the dapp I'm writing certainly won't need - like an ASN1 parser, an elliptic curve cyper, a 160k promise wrapper (wat) and the distributed filesystem libraries.

Have a browse of the contents of the compiled bundle.

Fixing this will not just lower download size, it will also make dapps load faster and improve the experience from nodejs. Node currently needs nearly 1 second to parse all that JS even on my fancy macbook:

$ node
> console.time('load'); require('web3'); console.timeEnd('load')
load: 750.081ms

[josephg]

... It also includes 2 copies of bn.js for some reason

[1blockologist]

Yeah, this is ridiculous. I'm preparing what is basically a static site and this is hugely problematic for many people worldwide that attempt access.

[williamchong]

as of beta34, I am having 6 bn.js of 2 different versions (4.11.6, 4.11.8) just by installing web3 please advise

[pubkey]

It looks like in beta.34, all submodules use the same bn.js@4.11.6. Maybe we can figure out why npm will install bn.js into each module instead of once. Webpack will then also only use bn.js once.

[pubkey]

The next step would be to move to es6-modules so we do not require the whole bn.js or underscore each time. Or is there already a plan for that?

[nivida]

Yes we have an 1.0ES6 branch @pubkey please feel free to contribute. I will have a look on the dependencies.

[roshanr95]

@nivida Is there a status page or something summarizing the current status of the ES6 branch? I can contribute, but don't really know where to start or what has been done so far.

Maybe a Github Project tracking it would be helpful.

[sweetpalma]

+1 to this issue. Final size of my bundle instantly bloated from 240kb to more than 1mb just by importing web3, analyzer stats are similar to @pubkey - six imports of bn.js. This is absolutely unacceptable, is there any workaround for this issue? I am using beta 35.

[nivida]

Hay all, I'm tested it with severall webpack configs for optimizing the package dependencies and I also checked the BN.js versions but with no success. The problem is that we have X package.json files for each web3 package this gives us the possibility to publish each package as standalone package on NPM but on the other hand it blows up the dependency graph. My current workarround would be to include the minified version of web3 directly from the node_modules dist folder this file is only ~700KB. I'll talk about this with fabian maybe we could change the project structure.

[pubkey]

I think it would be better to switch to a mono-package instead of publishing web3-eth web3-net etc as separate npm-modules. To me it does not make sense or will be possible to ever use different version of the sub-packages, so why not just publish web3 and the user can then require('web3/eth') and so on.

[ondratra]

@pubkey I agree with this.

Is there any progress on this issue? I am willing to help.

[josephg]

The problem is that we have X package.json files for each web3 package this gives us the possibility to publish each package as standalone package on NPM but on the other hand it blows up the dependency graph.

If you reference a version range in package.json, npm can reuse the same bn install across all dependancies. Eg, "bn.js"="^4.11.8" (note the ^) will use any version of bn from version 4.11.8 up to version 5. If you do it that way, the dependancy graph won't necessarily need multiple copies of bn.

[johnhforrest]

What's the current status of this? The bundle size is continuing to grow. beta36 added almost 200KB minified to the bundle size. At this point we are punting upgrading because of the extra bloat.

beta36 - 791.88 KB beta35 - 616.12 KB beta34 - 582.78 KB

I think your comment on Aug 20 is the correct way to solve the BN.js duplication issue. Maybe that can be done for just the purposes of building the dist files if it's there's too much risk to make a sweeping change like this.

[leonprou]

@pubkey @nivida running npm ls | grep bn.js you can see that most of the dependencies are deduped (not duplicated). But there are some dependencies that define a strict version of bn.js, they are:

• https://github.com/SilentCicero/number-to-bn
• https://github.com/ethjs/ethjs-unit
• web3-utils ( I guess it's doing this because of the two above)

So we should ask these packages to loosen their dependency to ~4.11.6 for example

[leonprou]

Different but related thing I would like to ponder about is how to make the web3 library more slim by reducing web3 packages I don't use. For example probably you need just one provider (http, websocket, ipc), not all of them. I think the same goes for many libraries.

So package.json could look like this:

    "web3-slim": "^1.0.0-beta.37"
    "web3-providers-http": "^1.0.0-beta.37"

web3-slim is the module that will assure the same usage of web3 module for coding. But in fact this module will include only the specified dependencies.

[pubkey]

@josephg yes, asking them to loosen their bn.js version would be a great improvement. About splitting web3 into more sub-packages, I'm not that sure. What you really want to have is that you can just import parts of Web3 and then your webpack or whatever bundling you to, should be able to tree-shake away the things you do not need. Also see my comment above. Splitting into more submodules would only be helpfull if you ever wanted to use different versions of parts of web3 which I do not think you need.

[leonprou]

@pubkey ok, for now the webpack is not working at all - https://github.com/ethereum/web3.js/issues/1969.

After this issue will be fixed we can get a look how the tree shaking is working. It's for sure the best option to get size improvements out of the box.

[nivida]

All of these problems will be solved after I got the PR #2000 merged :) @leonprou

[williamchong]

with beta.46, I am still getting 3 bn.js, and there are 4 lodash now edit: also somehow the umd version was included instead of esm, but other lib I used (vue) has esm included no problem in the same build

[BigMurry]

can use webpack alias to solve this issue.

1. publish a forked version of bn.js
2. yarn add fork-bn.js
3. change the webpack config

   module.exports = {
   // other configs
   resolve:{
   // other configs
   alias:{
     "bn.js":"fork-bn.js"
   }
   }
   }

   There is only one bn.js now, cheers :)

[nivida]

I've created a webpack example here and I think this is a good boilerplate to start. The long term goal will be to refactor the folder and module structure for preventing the multiple bundling of dependencies.

[BigMurry]

@nivida you list a better option

[levino]

This should not be part of a first stable release for 1.0. Not important.

[1blockologist]

@levino it should be fixed and work in progress but not hold up an actual stable release

So just fix which milestone its in and still get someone working on it?

[nivida]

This is in-progress because I've optimized the bundle size already a bit with the last refactoring. I think this is a duplication of https://github.com/ethereum/web3.js/issues/2109 because the simplifying of the project structure will solve this.

[adrianmcli]

As an update to the duplicate bn.js issue, Web3.js when installed right now will include two versions: 4.11.6 and 4.11.8.

That's because 4.11.6 is a pinned dependency in ethjs-unit and number-to-bn. Everywhere else, it seems like the versions line up so that's good.

I've made PRs to update them here:

https://github.com/ethjs/ethjs-unit/pull/2 https://github.com/SilentCicero/number-to-bn/pull/2

See npm ls bn.js output of Web3.js here:

➜  temp-web3 npm ls bn.js
temp-web3@1.0.0 /Users/adrianli/dev/temp-web3
└─┬ web3@1.2.2
  ├─┬ web3-bzz@1.2.2
  │ └─┬ swarm-js@0.1.39
  │   └─┬ eth-lib@0.1.27
  │     └── bn.js@4.11.8  deduped
  ├─┬ web3-eth@1.2.2
  │ ├─┬ web3-eth-abi@1.2.2
  │ │ └─┬ ethers@4.0.0-beta.3
  │ │   ├── bn.js@4.11.8  deduped
  │ │   └─┬ elliptic@6.3.3
  │ │     └── bn.js@4.11.8  deduped
  │ ├─┬ web3-eth-accounts@1.2.2
  │ │ ├─┬ crypto-browserify@3.12.0
  │ │ │ ├─┬ browserify-sign@4.0.4
  │ │ │ │ ├── bn.js@4.11.8  deduped
  │ │ │ │ ├─┬ browserify-rsa@4.0.1
  │ │ │ │ │ └── bn.js@4.11.8  deduped
  │ │ │ │ └─┬ parse-asn1@5.1.5
  │ │ │ │   └─┬ asn1.js@4.10.1
  │ │ │ │     └── bn.js@4.11.8  deduped
  │ │ │ ├─┬ create-ecdh@4.0.3
  │ │ │ │ └── bn.js@4.11.8  deduped
  │ │ │ ├─┬ diffie-hellman@5.0.3
  │ │ │ │ ├── bn.js@4.11.8  deduped
  │ │ │ │ └─┬ miller-rabin@4.0.1
  │ │ │ │   └── bn.js@4.11.8  deduped
  │ │ │ └─┬ public-encrypt@4.0.3
  │ │ │   └── bn.js@4.11.8  deduped
  │ │ ├─┬ eth-lib@0.2.7
  │ │ │ └── bn.js@4.11.8  deduped
  │ │ └─┬ ethereumjs-tx@2.1.1
  │ │   └─┬ ethereumjs-util@6.1.0
  │ │     ├── bn.js@4.11.8  deduped
  │ │     ├─┬ rlp@2.2.4
  │ │     │ └── bn.js@4.11.8  deduped
  │ │     └─┬ secp256k1@3.7.1
  │ │       └── bn.js@4.11.8  deduped
  │ └─┬ web3-eth-iban@1.2.2
  │   └── bn.js@4.11.8  deduped
  └─┬ web3-utils@1.2.2
    ├── bn.js@4.11.8
    ├─┬ eth-lib@0.2.7
    │ ├── bn.js@4.11.8  deduped
    │ └─┬ elliptic@6.5.1
    │   └── bn.js@4.11.8  deduped
    ├─┬ ethjs-unit@0.1.6
    │ └── bn.js@4.11.6
    └─┬ number-to-bn@1.7.0
      └── bn.js@4.11.6

[nivida]

@adrianmcli Oh thanks!

[josephg]

Thanks everyone for helping. I'm sure removing all the extraneous instances of bn.js has helped a lot. Coming up on the 2 year anniversary of this bug, I was curious how metrics look compared to when I opened this issue, so I re-ran my earlier test. Note, for a like-to-like comparison I am again using pre-minified sizes. With minification the numbers halve, like they did 2 years ago. But the bundle is minified by default if you install from npm now, which is nice.

To avoid minification I manually checked out web3.js into node_modules/web3, then ran npm i from there.

$ cat foo.js 
require('web3')
$ npx browserify foo.js -o bundle.js
npx: installed 138 in 4.206s
$ ls -l bundle.js 
-rw-r--r--  1 josephg  staff  2022827  9 Nov 10:00 bundle.js
$ node
Welcome to Node.js v12.10.0.
> console.time('load'); require('web3'); console.timeEnd('load')
load: 728.931ms

So in 2 years instead of shrinking, the library has nearly doubled in size from 1.36MB to 2.02MB.

Loading the library in nodejs via npm on a $2000 laptop now only takes 729ms (down from 750ms!) which I suspect is because of optimizations in v8. With the minified version sitting in npm we get down to 510ms.

After 2 years I still can't use this in my web apps, because multi-second page loads on a phone are a non-starter. I'm still confused why the bundle includes what it does - like 2 copies of the elliptic library. (Especially since elliptic curve crypto is already part of the web crypto API and supported in all browsers). Clearly some effort has been made to split the module up, so maybe there's a way to use the parts independently? Some documentation on this would be great. All I want is to be able to call methods in a smart contract. Almost none of the code in web3.js is needed for this - I don't need to talk to ENS. I don't even know what bzz is - but I am pretty sure people loading my webpage don't need to be a peer in a distributed file swarm. I'm also pretty sure I don't need an asn.1 decoder.

The target should be a pre-minification size of about 200kb. Thats about on par with react js, and its about 10x smaller than the current bundle. Deduping bn.js helps, but it will not be enough on its own.

[1blockologist]

@josephg valid criticisms that I share, but what I ended up doing was just letting it be part of the web app anyway. You say 'cant', is that a real limitation for your users and use case? Is a loading screen or some other kind of distraction or even segmented loading not an option for you? You don't necessarily need the entire bundle immediately when a client loads your very first page.

[adrianmcli]

@josephg interesting stats. I wonder how it compares to Ethers.js?

[cgewecke]

(Cross referencing #3160 which has ongoing work to modernize the build.)

It would be nice to have an established way of benchmarking the performance issues in this thread so there's a common point of reference. (If anyone has ideas about that, please advise). Have conducted a basic experiment and am seeing the following.

With:

• Macbook Air 1.8ghz
• Chrome (cacheing disabled)
• Comcast download speed avg: 28.7 mbps
• Avg mobile download speeds (US, 2019)]: 33.88 mbps (Europe is ~15% slower)

  <!DOCTYPE html>
  <html>
  <head>
  <!-- Polyfill for web3 due to issue #3155 -->
  <script src="https://cdn.jsdelivr.net/npm/babel-polyfill/dist/polyfill.min.js"></script>
  <script src="https://cdn.jsdelivr.net/gh/ethereum/web3.js@1.2.1/dist/web3.min.js"></script>
  <script src="https://cdn.ethers.io/scripts/ethers-v4.min.js"></script>
  </head>
  <body>
  </body>
  </html>

  

So - there is a load time difference vs Ethers (1/3 the size), ~150ms, over CDN.

• Is there a better test than this?
• Is there a standard way to benchmark mobile load times?
• How significant is 150ms to UX on the front end? (@adrianmcli what's your sense of this?)

My Time to require("web3") MacBook Air 1.8ghz

$ time node require_web3.js

real    0m0.450s
user    0m0.454s
sys 0m0.057s  

Tbh most of my experience with Web3 is on the backend w/ JS testing & Solidity. Just guessing and take with a grain of salt but probably 80% of tasks last > 30s. Many tasks are in the 5 to 10 min range. There are places where a 50% performance gain overall would be life-changing for the engineers - here for example.

At those scales the optimizations inevitably lie elsewhere, although it's really nice when things start quickly.

[nivida]

@josephg I can recommend you to do such tests after we have merged the newly created build pipeline of PR #3160. Further bundle size improvements have to be checked closer after and we have to be sure to not break anything. The web3 package downloaded from NPM and bundled with zero-config webpack was before around 4.4mb and is now down to 1mb which is clearly an improvement. Using of native crypto libs do gave us some compatibility problems in the past. web3.js and many other EF JS projects will soon use because of this the ethereum crypto packages @alcuadrado is creating.

We give our best to improve the bundle sizes and functionalities of web3.js step by step.

[williamchong]

@josephg valid criticisms that I share, but what I ended up doing was just letting it be part of the web app anyway. You say 'cant', is that a real limitation for your users and use case? Is a loading screen or some other kind of distraction or even segmented loading not an option for you? You don't necessarily need the entire bundle immediately when a client loads your very first page.

In my case I used web3 in a SPA webapp with SSR. The huge download and parsing time causes my page to freeze after html/css render for >3 seconds before user can actually interact with js button, even on flagship smartphone. In this case user might just think the site is broken and leave I end up using lazy load with webpack in pages where user need to perform eth signature to mitigate this problem

[josephg]

@nivida I saw that after I posted my reply. I think those changes will help a lot - I mean, realistically the right approach here is to only load the JS we need, and splitting web3 out into separate modules so I can include just those parts I need would fit the bill.

@cgewecke I mentioned the nodejs startup time because its annoying, but its the time to open the library in a browser I really care about. And the real number here isn't how long it takes to download, unzip and parse 1MB of JS on a good internet connection from the expensive laptop I use for development. The number I care about is how it impacts page load times on my friends' crappy android phone when we're at a coffee shop. Last time I checked phones still only parsed JS about 1/8th the speed of desktop CPUs. By that ratio 241ms in chrome is probably more like 2 seconds on a cheap phone; which is more than long enough for users to assume my webpage is broken and close the tab. (And remember, this is one of many javascript libraries I use in web apps. I need my total page time on a phone to be ideally below 500ms. I don't want to blow that budget on a single library.)

Desktop speed still matters though. I disabled metamask a couple years ago because it loaded web3 in each of my browser tabs, and that was making chrome about 250ms slower every single time I clicked a link or opened a tab. It was a really noticeable change when I turned metamask off.

But yeah; it would be great to have a standard speed metric to focus on. Maybe minified size (pre-gzip)? I'd expect parse time to be roughly linear with minified size across all devices.

In any case, I think the low hanging fruit here is removing all the web3.js features my app doesn't care about from the bundle. For me this includes anything not relevant to calling smart contract functions. Looking at the bundle contents, I think that would improve things by a factor of 3x or so.

We're obviously a long way from this point, but my ideal implementation would include a build process which looked at my smart contract, parsed it, then made a specialized JS bundle with method calls for each of my smart contract functions. Since webcrypto is supported on all browsers I care about, I doubt we'd need more than 100k for something like that - and then only because bn.js is 85k. Should be less once BigInt is finally supported in Edge and Safari.

[cgewecke]

Some more benchmark resources for this, testing minimal-case mobile loads on 7 year old phones with dotcom-monitor.

Baseline check: published to surge.sh. NB: looks like surge introduces an additional 15-20% of blocking/overhead - could be less though, (see chrome developer window below)

<!DOCTYPE html>
<html>
  <head>
    <!-- Polyfill for web3 due to issue #3155 -->
    <script src="https://cdn.jsdelivr.net/npm/babel-polyfill/dist/polyfill.min.js"></script>
    <script src="https://cdn.jsdelivr.net/gh/ethereum/web3.js@1.2.1/dist/web3.min.js"></script>
  </head>
  <body>
    <h1> Loaded... </h1>
  </body>
</html>

Android / Nexus 4: (Released 2012 - Discontinued 2013)

IPhone 5 (Released 2012 - Discontinued 2013)

Other resources

• 2018 Google study on mobile load times. It looks like they think best practice is 1s - 3s initial render, everything painted by 500kb and are seeing avg total resource loads for mobile apps > 20s.

[slavafomin]

Oh my god, this is just crazy. I've updated web3js package in my old application to the latest version and it had ridiculously grown in size. The package takes 1.12 MB, which accounts for 35% of the pretty big application!

However, I'm using only this functionality:

• eth.defaultAccount
• eth.sendTransaction
• utils.toWei
• utils.toHex
• utils.isAddress

I've tried to use imports like these:

import { Eth } from 'web3-eth';
import { isAddress, toHex, toWei } from 'web3-utils';

It helped to reduce the size a little, but not dramatically. And I can see that bn.js and sha3 are both duplicated three times.

All I want is to let users to send transactions using Metamask, this is a small portion of the app, but the library takes tremendous amount of space. I guess the only option right now is to put it into separate bundle and load it dynamically when needed, otherwise it will blow up the application loading time.

Are you planning to release the code in ESM format so it could be tree shaken?

Right now I can see that you are using CommonJS with imports like this: var _ = require('underscore');, which is extremely ineffective because it causes entire libraries to be loaded, instead of just the needed parts, rendering tree shaking instruments useless.

[slavafomin]

[nivida]

@slavafomin This was sadly always the case. We do currently improve the build pipeline which does also change the module structure from CJS to ESM (#3160). Further improvements will be done after.

[slavafomin]

@nivida that's a great news, thank you. I hope it will make things better.

[cgewecke]

(Not stale.)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. If you believe this was a mistake, please comment.

[josephg]

Not stale.

[epiqueras]

This is abysmal...

[GregTheGreek]

Update:

After discussions with the Swarm team we are removing bzz support #3746

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. If you believe this was a mistake, please comment.

[josephg]

Not stale

[josephg]

3 year anniversary benchmark:

{11:37}~/temp ➭ mkdir web3
{11:37}~/temp ➭ cd web3
{11:37}~/temp/web3 ➭ npm i web3
...
+ web3@1.3.1
{11:38}~/temp/web3 ➭ du -hs node_modules 
 53M    node_modules
{11:38}~/temp/web3 ➭ find . | wc -l
    7649
{11:39}~/temp/web3 ➭ cat > foo.js
require('web3')
{11:39}~/temp/web3 ➭ browserify foo.js -o bundle.js
{11:39}~/temp/web3 ➭ ls -l
total 6384
-rw-r--r--    1 josephg  staff  2379951 Dec 24 11:39 bundle.js
...
{11:42}~/temp/web3 ➭ terser -cm < bundle.js | wc -c
    1156777
{11:42}~/temp/web3 ➭ terser -cm < bundle.js | gzip -c | wc -c
    324633
{11:40}~/temp/web3 ➭ node
Welcome to Node.js v14.15.1.
> console.time('load'); require('web3'); console.timeEnd('load')
load: 775.534ms
{11:46}~/temp/web3 ➭ npm ls | grep ' bn.js@' 2> /dev/null
  │ │ │ ├── bn.js@4.11.9 deduped
  │ │ │ │ ├── bn.js@4.11.9 deduped
  │ │ │ │ └── bn.js@4.11.9 deduped
  │ │ │ │ ├── bn.js@5.1.3 extraneous
  │ │ │ │ │ ├── bn.js@5.1.3 extraneous
  │ │ │ │ │ │ ├── bn.js@4.11.9 deduped
  │ │ │ │ ├── bn.js@4.11.9 deduped
  │ │ │ │ ├── bn.js@4.11.9 deduped
  │ │ │ │ │ ├── bn.js@4.11.9 deduped
  │ │ │ │ ├── bn.js@4.11.9 deduped
  │ │ │   ├── bn.js@4.11.9 deduped
  │ │ │     └── bn.js@4.11.9 deduped
  │ │ ├── bn.js@4.11.9 deduped
    ├── bn.js@4.11.9
    │ ├── bn.js@4.11.6 extraneous
    │ ├── bn.js@4.11.6 extraneous

So:

• Bundle size (with browserify) has grown from to 2.3mb uncompressed, 1.1mb minified and 324kb minified and gzipped. Which is still so large as to be hilariously unusable.
• Nodejs startup time is more or less unchanged (775ms up from 730ms). So web3's size is more or less keeping pace consuming v8's continual performance improvements
• I have no idea how many of those copies of bn.js make it into the final build. All of them are extraneous at this point because BigInt is supported everywhere now.
• Just installing web3 in npm takes ~20 seconds, and takes up 53 megs on disk across 7800 files.

I look forward to future anniversaries of this issue.

[GregTheGreek]

I look forward to future anniversaries of this issue.

Can't wait to prove you wrong next year!

If you don't already know this project has been recently taken over by ChainSafe, so please give us some time, we're prioritising a few other things. We're working are way through a modernisation effort, and tackling the bundle size will come right after.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. If you believe this was a mistake, please comment.

[josephg]

Not stale