search

cowprotocol / dune-bridge

Other
4 stars 3 forks source link

Parallelize IPFS Requests for App Data #2

Open nlordell opened 2 years ago

nlordell commented 2 years ago

This iterative async block of code could probably be a batch request.

    for hash in uninitialized_app_data_hashes.iter() {
        download_referral_from_ipfs_and_store_in_referral_store(db.clone(), *hash).await?;
    }

cf referral_maintenance.rs

Original issue https://github.com/gnosis/gp-dune-bridge/issues/59 by @bh2smith

josojo commented 2 years ago

Yes, this should be doable. Just a word of warning: When I was first implementing it, I had to fight with rate limitings. If we parallelize it, there is a risk that this will reappear

bh2smith commented 2 years ago

To avoid the rate limiting (maybe) You can put a cap on the size of the batches.