Modify config.py with new prompts, coins & error tolerances
Run pip install -r requirements.txt
Run python benchmarks.py price for price benchmarks
Run python benchmarks.py mcap for market cap benchmarks
Considerations
It runs very slow because of coingecko rate limits. Disabling coingecko adapter lets us reduce the time.sleep() in benchmarks.py and a little bit faster but the data agent can still be rate limited.
The source of truth asset id is the coingecko id. Any new adapters will need some way of translating the coingecko id if they use something else. For example, the defillama adapter "just works" because they use coingecko ids.
Measures data agent price & market cap responses versus benchmark values from Coingecko, Defillama, etc.
See config.py for prompt & coin list templates
Running:
cd submodules/benchmarksconfig.pywith new prompts, coins & error tolerancespip install -r requirements.txtpython benchmarks.py pricefor price benchmarkspython benchmarks.py mcapfor market cap benchmarksConsiderations
It runs very slow because of coingecko rate limits. Disabling coingecko adapter lets us reduce the
time.sleep()inbenchmarks.pyand a little bit faster but the data agent can still be rate limited.The source of truth asset id is the coingecko id. Any new adapters will need some way of translating the coingecko id if they use something else. For example, the defillama adapter "just works" because they use coingecko ids.