Let's say I want to download all the uniswap v2 pairs with their reserves. One will do the following:
1- create the uniswap v2 contract object query, then contract.queryAllPairsLength
2- foreach i in AllPairsLength contract.query allPairs(i) => this returns an address
2.1 - create a uniswapv2Pair contract using the address returned
2.2 - query the contract for functions token0 and token1 and getReserves
given there are 70k+ pairs, this can be done concurrently, one can simply create a create 10+ threads to concurrently fetch 1000 address per batch and for each of them create batch a batch to query 2.2
I have done something similar to what is described above in both golang (using github.com/lmittmann/w3) and this crate in rust but I have noticed that the rust version is at least 10x slower than golang. Both are running in the same machine query an erigon full node.
I have done the following approaches to speed things up:
1- create a single WebSocket and create a web3<batch> and cloned it in every tokio threac
2- created a single WebSocket to get the AllPairsLength but created new WebSocket object inside every thread to have a unique websocket connection. I create this in step 2- of the pseudo code above.
option 2 was even slower.
I am not sure why there is such a large performance difference between my rust implementation and golang implementation and I figured I may be using this crate in the wrong way.
Can you provide guidance as to whether we should use the same websocket or create a new websocket for faster performance?
Any tricks to speed things up.
Appreciate your help and apologies for posting it here as I am not aware of another medium to share this
Let's say I want to download all the uniswap v2 pairs with their reserves. One will do the following: 1- create the uniswap v2
contractobject query, thencontract.queryAllPairsLength2-foreach i in AllPairsLengthcontract.queryallPairs(i)=> this returns an address 2.1 - create a uniswapv2Pair contract using the address returned 2.2 - query the contract for functionstoken0andtoken1andgetReservesgiven there are 70k+ pairs, this can be done concurrently, one can simply create a create 10+ threads to concurrently fetch 1000 address per batch and for each of them create batch a batch to query 2.2I have done something similar to what is described above in both golang (using github.com/lmittmann/w3) and this crate in rust but I have noticed that the rust version is at least 10x slower than golang. Both are running in the same machine query an erigon full node. I have done the following approaches to speed things up: 1- create a single> and cloned it in every tokio threac
2- created a single WebSocket to get the AllPairsLength but created new WebSocket object inside every thread to have a unique websocket connection. I create this in step 2- of the pseudo code above.
WebSocketand create a web3<batchoption 2 was even slower. I am not sure why there is such a large performance difference between my rust implementation and golang implementation and I figured I may be using this crate in the wrong way. Can you provide guidance as to whether we should use the same websocket or create a new websocket for faster performance? Any tricks to speed things up.
Appreciate your help and apologies for posting it here as I am not aware of another medium to share this