search

overlay-market / overlay-risk

Risk metrics for various data streams
MIT License
13 stars 4 forks source link

Code for back-filling ovl_sushi bucket #15

Closed deepsp94 closed 3 years ago

deepsp94 commented 3 years ago

@mikeyrf, The script "backfill_influx_bulk.py" is just a quickly hacked together. These are the steps it goes through:

  1. Read the last 30 days of data from your ovl_sushi bucket
  2. Fix token0/1 names where required
  3. Pivot the data to upload easily into my ovl_sushi bucket
  4. The upload into my bucket has been broken down into batches of 2000 rows instead of 1 big upload. Had to do this to avoid the error "max write limit exceeded"

Let me know if you need a more parameterized script (like influx_csv.py) and I'll make the changes.

mikeyrf commented 3 years ago

@deepsp94 yea I think a more parametrized script like influx_csv.py would be nice here. I think it would be interesting to specify via command line flags:

Along with the standard params, config setup we have going. Let me know if this works for you.

Made some small cleanup changes as well so github actions passes successfully (flake8)

deepsp94 commented 3 years ago

Hey @mikeyrf, i'm working on the other issues too, just wanted to get this merged in without any conflicts. Lemme know if you get a chance to review :)

mikeyrf commented 3 years ago

Hey @mikeyrf, i'm working on the other issues too, just wanted to get this merged in without any conflicts. Lemme know if you get a chance to review :)

Sounds good! Let me review real quick

mikeyrf commented 3 years ago

Looking great :) ... merge whenever you like