Coingecko ETH Price API delay

vintrocode commented 2 years ago

We might want to add some exception handling to the jobs/update_eth_prices.py because the entire job fails if the coingecko API isn't up-to-date with ETH pricing data.

Today's September 11th. Coingecko has prices for the 9th, but not the 10th or 11th. This throws an error at line 48 of update_eth_prices.py. An example return on a failure day:

{"id":"ethereum","symbol":"eth","name":"Ethereum","localization":{"en":"Ethereum","de":"Ethereum","es":"Ethereum","fr":"Ethereum","it":"Ethereum","pl":"Ethereum","ro":"Ethereum","hu":"Ethereum","nl":"Ethereum","pt":"Ethereum","sv":"Ethereum","vi":"Ethereum","tr":"Ethereum","ru":"Эфириум","ja":"イーサリアム","zh":"以太坊","zh-tw":"以太幣","ko":"이더리움","ar":"يثريوم","th":"Ethereum","id":"Ethereum","cs":"Ethereum","da":"Ethereum","el":"Ethereum","hi":"Ethereum","no":"Ethereum","sk":"Ethereum","uk":"Ethereum","he":"Ethereum","fi":"Ethereum","bg":"Ethereum","hr":"Ethereum","lt":"Ethereum","sl":"Ethereum"},"image":{"thumb":"https://assets.coingecko.com/coins/images/279/thumb/ethereum.png?1595348880","small":"https://assets.coingecko.com/coins/images/279/small/ethereum.png?1595348880"}}

No market_data key. Any suggestions for how to handle?

dmatsuoka commented 2 years ago

@vintrocode Thanks for sharing. I'm not seeing any issues with the CoinGecko responses. Are you are making the request using the 'dd-mm-yyyy' date format?

https://api.coingecko.com/api/v3/coins/ethereum/history?date=11-09-2022

However, I think there is an issue with exceeding the free tier rate limits. We can increase the delay between requests, which I think will resolve this.

vintrocode commented 2 years ago

@dmatsuoka thanks for the quick response

my manual checks of the API were indeed using 'mm-dd-yyyy' format, so i was explaining why this was failing incorrectly.

now i'm not sure what's causing it to fail. i don't have any control over the date input format and the code looks correct. i've tried following the end-to-end example with a fresh install with the same problem...

git clone https://github.com/a16z/nft-analyst-starter-pack && cd nft-analyst-starter-pack
poetry env use python3.9
poetry install
poetry shell

(nft-analyst-starter-pack-WJgYzHZM-py3.9) ➜  nft-analyst-starter-pack git:(main) python export_data.p
y -a $ALCHEMY_API_KEY -c 0xBd3531dA5CF5857e7CfAA92426877b022e612cf8
Process started for contract address: 0xBd3531dA5CF5857e7CfAA92426877b022e612cf8
Checking update logs for most recent block...
No existing data. Contract 0xBd3531dA5CF5857e7CfAA92426877b022e612cf8 appears to have been deployed at block 12876179
Exporting token transfers...
Exporting logs...
Updating block-to-date mapping...
Updating ETH prices...
Traceback (most recent call last):
  File "/Users/vintrocode/work/passage/pudgy/nft-analyst-starter-pack/export_data.py", line 225, in <module>
    export_data()
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-WJgYzHZM-py3.9/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-WJgYzHZM-py3.9/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-WJgYzHZM-py3.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-WJgYzHZM-py3.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/vintrocode/work/passage/pudgy/nft-analyst-starter-pack/export_data.py", line 173, in export_data
    update_eth_prices(filename=eth_prices_csv)
  File "/Users/vintrocode/work/passage/pudgy/nft-analyst-starter-pack/jobs/update_eth_prices.py", line 48, in update_eth_prices
   price_of_eth = j["market_data"]["current_price"]["usd"]
KeyError: 'market_data'

are you able to replicate?

dmatsuoka commented 2 years ago

@vintrocode I believe it's rate limits. Can you try changing line 36 in update_eth_prices.py from sleep(2) to sleep(5) and see if it fixes it?

vintrocode commented 2 years ago

that was it. had to go with sleep(10) though. thanks for the help! @dmatsuoka

a16z / nft-analyst-starter-pack

Coingecko ETH Price API delay #20