zelos-alpha / demeter-fetch

On chain data download tool for demeter
https://medium.com/zelos-research
Apache License 2.0
12 stars 9 forks source link
aave-v3 defi ethereum google-cloud uniswap-v3

README

1. Introduction

dDemeter-fetch is a member of the demeter family, which is responsible for fetching and processing on-chain data. It provides standardized data for backtesting and defi research.

graph TD;
   demeter-fetch([demeter-fetch]) --> demeter([demeter]);
   demeter-fetch --> DeFi-research([DeFi research]);

Demeter-fetch can fetch chain event logs of DEFI projects from various sources and convert them into a unified format. In the project dimension, it supports Uniswap and AAVE. In the data source dimension, the supported data sources include:

Demeter-fetch supports export formats such as:

The minutely OHLC is mainly used in Demeter, while tick data is used in research, where being transformed into many forms.

Demeter-fetch uses streaming processing. It will summarize all the channels into raw files first, and then transform them into various forms. The processing flow is shown in the figure, in which green blocks are finished, and yellow blocks are to-do.

flowchart TB;
   RPC:::done --rpc request--> raw:::done;
   BigQuery:::done --api request--> raw;
   Chifra:::done --cli export--> raw;
   raw --OHLC--> minute:::done;
   raw --formatted transaction--> tick:::done;
   raw --> Squeeth:::done;
   tick --> Uniswap;
   tick --> AAVE;
   tick --> other_defi;
   subgraph Uniswap;
      direction TB;
      position:::done;
      address_LP[LP of address]:::done;
      fee_on_tick[fee on tick]:::to_do;
      return_rate[return rate]:::to_do;
   end
   subgraph AAVE;
      aave_tick[tick]:::done;
   end
   subgraph other_defi;
      other_tick[tick]:::to_do;
   end
   classDef done fill:#00ff00
   classDef to_do fill:#ffff00

2 design rationale

  1. not node not your data. Demeter fetch does not directly provide data, you need config your data source.
  2. config,run, done! use it without knowledge like event log, solidity,ethereum. Keep it simple.
  3. customized data output. from tick to address net value. Do your own research with various data.

3 Use cases:

chaoslab research: At this juncture, we retrieve all on-chain swaps for the selected pool over a span of 180 days, enabling us to replay them during the backtesting phase. This data retrieval is facilitated via Google BigQuery, which is employed by Demeter intrinsically. We extend our gratitude to the Demeter team for the creation and open-sourcing of this valuable library to the wider Uniswap community.

zelos research It is our own research. Demeter fetch have us won first prize in Bounty #20 of UGPCA

4 Roadmap

5 Links