[Lake] Parallelize fetching data across >>1 coins

trentmc commented 2 months ago

Background / motivation

We currently fetch data from Binance one token at at time.

So if there are 10 coins (BTC, ETH, DOT, ..) then it takes 10x longer.

TODOs / DoD

[ ] Parallelize fetching data, in lake/ohclv_data_factory.py. It's likely a straightforward conversion of a for loop

Related github issues

#804 "[Sim, make $] Make benchmarking parallel, via threading"

Note: we could also parallelize grabbing data within a feed. However this isn't as easy, because the algorithm needs to check what's there already. So consider this for later. But also maybe not needed because at some point we'll have historical data repo / bundle.

trentmc commented 2 months ago

cc @calina-c @idiom-bytes

calina-c commented 2 months ago

I don't recommend doing this now, since the structure is changing for lake either way. It will only result in either difficult conflicts or entirely lost while fixing said conflicts. I agree it is a good thing, but we should wait until the lake/ETL part is done.

trentmc commented 2 months ago

I don't recommend doing this now, since the structure is changing for lake either way. It will only result in either difficult conflicts or entirely lost while fixing said conflicts. I agree it is a good thing, but we should wait until the lake/ETL part is done.

OK. Makes sense.

oceanprotocol / pdr-backend