Open WilfredAlmeida opened 1 month ago
Very nice you fixed all my issues I was having when subscribing to a load of accounts on program startup...
I use this
export const connection = new Connection(
config.rpc.httpEndpoint,
{
httpAgent: new Agent({
keepAlive: config.rpc.keepAlive,
keepAliveMsecs: 60000,
maxSockets: 256,
}),
...config.rpc,
fetch: RetryFetcher,
},
);
I wrote benchmarks to find this number!
git clone https://github.com/solana-labs/solana-web3.js
cd solana-web3.js/packages/fetch-impl/
pnpm i
pnpm benchmark https://some.rpc-server-with-no-rate-limit.com
> @solana/fetch-impl@0.0.0 benchmark /home/sol/src/solana-web3.js-git/packages/fetch-impl
> ./src/__benchmarks__/run.ts "https://some.rpc-server-with-no-rate-limit.com"
(node:2243087) [UNDICI-H2] Warning: H2 support is experimental, expect them to change at any time.
(Use `node --trace-warnings ...` to show where the warning was created)
┌─────────┬───────────────────────────────────┬─────────┬────────────────────┬───────────┬─────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ns) │ Margin │ Samples │
├─────────┼───────────────────────────────────┼─────────┼────────────────────┼───────────┼─────────┤
│ 0 │ 'no dispatcher' │ '0' │ 1_902_934_963 │ '±37.13%' │ 10 │
│ 1 │ 'unlimited connections http/2' │ '0' │ 4_605_433_659 │ '±1.58%' │ 10 │
│ 2 │ 'unlimited connections' │ '0' │ 4_043_684_846 │ '±4.88%' │ 10 │
│ 3 │ '16 connections' │ '0' │ 6_892_149_604 │ '±41.75%' │ 10 │
│ 4 │ '16 connections, http/2' │ '2' │ 439_921_781 │ '±21.03%' │ 10 │
│ 5 │ '16 connections, pipeline 2 wide' │ '0' │ 2_944_779_337 │ '±17.62%' │ 10 │
│ 6 │ '32 connections' │ '0' │ 4_135_789_213 │ '±51.36%' │ 10 │
│ 7 │ '32 connections, http/2' │ '1' │ 681_074_594 │ '±39.77%' │ 10 │
│ 8 │ '32 connections, pipeline 2 wide' │ '0' │ 3_762_177_439 │ '±38.15%' │ 10 │
│ 9 │ '64 connections' │ '0' │ 2_845_502_630 │ '±27.39%' │ 10 │
│ 10 │ '64 connections, http/2' │ '1' │ 784_407_918 │ '±32.93%' │ 10 │
│ 11 │ '64 connections, pipeline 2 wide' │ '0' │ 2_871_394_050 │ '±17.78%' │ 10 │
└─────────┴───────────────────────────────────┴─────────┴────────────────────┴───────────┴─────────┘
So at least in my testing:
1
is the right number, but unlimited
doesn't seem great either.@WilfredAlmeida, can you download that benchmark suite, add in some tests (connections down to 1, above 64, and add in some wider pipelines) and run them against Triton servers? I'd do it but I'm pretty sure I've already been blocklisted on Mainnet.
Finally got a clean test run for 1 and 128 connections.
┌─────────┬────────────────────────────────────┬─────────┬────────────────────┬───────────┬─────────┐
│ (index) │ Task Name │ ops/sec │ Average Time (ns) │ Margin │ Samples │
├─────────┼────────────────────────────────────┼─────────┼────────────────────┼───────────┼─────────┤
│ 0 │ '1 connections' │ '0' │ 294_520_572_463 │ '±0.05%' │ 10 │
│ 1 │ '1 connections, pipeline 2 wide' │ '0' │ 294_785_419_969 │ '±0.16%' │ 10 │
│ 2 │ '1 connections, pipeline 8 wide' │ '0' │ 171_703_417_929 │ '±38.19%' │ 10 │
│ 3 │ '128 connections' │ '0' │ 3_147_103_325 │ '±2.11%' │ 10 │
│ 4 │ '128 connections, pipeline 2 wide' │ '0' │ 3_088_783_513 │ '±1.10%' │ 10 │
│ 5 │ '128 connections, pipeline 8 wide' │ '0' │ 3_093_144_608 │ '±0.99%' │ 10 │
└─────────┴────────────────────────────────────┴─────────┴────────────────────┴───────────┴─────────┘
undici
dispatcher configs and let people plug them in to web3.js. cc/ @0xIchigo @johnpmitschSounds good to me
It'd be better if we could have it mentioned in the web3.js docs something like "If you're facing socket issues, based on your RPC provider, check this out"
We already have the doc published here
At Triton One, we have observed our customers face socket/connection issues. More information here
These issues have been there for more than 2 years.
tl;dr of the issues is that NodeJS tries to open infinite connections with the server and hits a limit and existing connections break and new connection attempts time out. In our tests, NodeJS opened over 16,000 connections. Opening new connection is unnecessary since existing ones can be resued. This is not the case with the Bun runtime. It opens less than 400 connections and doesn't error out at all.
With the current versions of web3.js (latest 1.91.8 as of this writing) the issues are frequent. We've also tested with the new web3.js
2.0.0-preview.3
and the error rate is less, but it still does exist.The fix for these issues, as mentioned in the doc shared above, is to limit the number of connections NodeJS can open. This can be configured with the
http.Agent
configurations.We propose that Web3JS should have a default configurable connection limit like the following:
From the
http.Agent
docsSetting a default limit will allow connection reuse instead of a new connection being set for every request.