search

zhehaowang / sneaky

Feed and strategy for cross-venue Sneakers trading (Du, StockX).
GNU Lesser General Public License v3.0
10 stars 3 forks source link

feedv2: DB-ify #52

Open zhehaowang opened 4 years ago

zhehaowang commented 4 years ago

Updating / reading 20K files in 1.6K folders and each time decoding and encoding the whole file might be inefficient at some point (!). We should look into transitioning to a DB-based (data warehouse / columnar) solution.

The hope is that the barrier was strong enough to ease this transition.

Should we do this before or after cloud deployment?

zhehaowang commented 4 years ago

Probably the best approach will be two / three csvs for each?