search

daoswapdex / docs

0 stars 0 forks source link

Query Uniswap data and save in db #5

Open daodaoswap opened 3 years ago

daodaoswap commented 3 years ago

Need use Java to do below task:

  1. Query uniswap data in https://v2.info.uniswap.org/#/
  2. Transfer these data to structure like subgraph, save to a database
  3. So that I can query anytime, for example I can query the swap value of one or many wallet by wallet address, include transfer time/amount etc.

Deliveries requirement:

  1. Use Java
  2. Need to describe the solution
  3. And delivery code that can be used directly.
gitcoinbot commented 3 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 300.0 DAI (300.0 USD @ $1.0/DAI) attached to it.

gitcoinbot commented 3 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Workers have applied to start work.

These users each claimed they can complete the work by 265 years, 2 months from now. Please review their action plans below:

1) controlcpluscontrolv has applied to start work _(Funders only: approve worker | reject worker)_.

So let me break down what I will do for you.

  1. Build a Java script that queries an Ethereum Node Endpoint, like Alchemy or Infura Node
  2. Using the Address of the Uniswap Factory, I can find all Uniswap Pairs and their addresses as well as coin composition.
  3. With that I have all tx's of Uniswap Pairs and trades, calling balanceOf I can get historical pool composition too!

Your requirement in actual data delivered was a bit hard for me to understand, so I will need further clarification on what schema and what data you actually wanted stored, but using the aforementioned techniques I can extrapolate all the data shown on Uniswap Analytic's site. I would then be able to store in a MongoDB for you to query and run.

I understand me not having Java in my portfolio might not be that inspiring, but I know the basics of the language to make queries and C(reate)R(remove)U(pdate)D(elete) in a database

Learn more on the Gitcoin Issue Details page.

ControlCplusControlV commented 3 years ago

Glad to be accepted! Would you mind providing me with a schema for the database you want made?

An example would be something like Collections for Each pair A Document for each day tracked inside of that collection containing all tx's with that pair

Just so I can make sure I am organizing my data how you want it!

gitcoinbot commented 3 years ago

@controlcpluscontrolv Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

ControlCplusControlV commented 3 years ago

I am awaiting a response on how @daodaoswap is hoping for me to structure my schema

daodaoswap commented 3 years ago

@con Please check below.

type UniswapFactory @entity {

factory address

id: ID!

pair info

pairCount: Int!

total volume

totalVolumeUSD: BigDecimal! totalVolumeETH: BigDecimal!

untracked values - less confident USD scores

untrackedVolumeUSD: BigDecimal!

total liquidity

totalLiquidityUSD: BigDecimal! totalLiquidityETH: BigDecimal!

transactions

txCount: BigInt! }

type Token @entity {

token address

id: ID!

mirrored from the smart contract

symbol: String! name: String! decimals: BigInt!

used for other stats like marketcap

totalSupply: BigInt!

token specific volume

tradeVolume: BigDecimal! tradeVolumeUSD: BigDecimal! untrackedVolumeUSD: BigDecimal!

transactions across all pairs

txCount: BigInt!

liquidity across all pairs

totalLiquidity: BigDecimal!

derived prices

derivedETH: BigDecimal

derived fields

tokenDayData: [TokenDayData!]! @derivedFrom(field: "token") pairDayDataBase: [PairDayData!]! @derivedFrom(field: "token0") pairDayDataQuote: [PairDayData!]! @derivedFrom(field: "token1") pairBase: [Pair!]! @derivedFrom(field: "token0") pairQuote: [Pair!]! @derivedFrom(field: "token1") }

type Pair @entity {

pair address

id: ID!

mirrored from the smart contract

token0: Token! token1: Token! reserve0: BigDecimal! reserve1: BigDecimal! totalSupply: BigDecimal!

derived liquidity

reserveETH: BigDecimal! reserveUSD: BigDecimal! trackedReserveETH: BigDecimal! # used for separating per pair reserves and global

Price in terms of the asset pair

token0Price: BigDecimal! token1Price: BigDecimal!

lifetime volume stats

volumeToken0: BigDecimal! volumeToken1: BigDecimal! volumeUSD: BigDecimal! untrackedVolumeUSD: BigDecimal! txCount: BigInt!

creation stats

createdAtTimestamp: BigInt! createdAtBlockNumber: BigInt!

Fields used to help derived relationship

liquidityProviderCount: BigInt! # used to detect new exchanges

derived fields

pairHourData: [PairHourData!]! @derivedFrom(field: "pair") liquidityPositions: [LiquidityPosition!]! @derivedFrom(field: "pair") liquidityPositionSnapshots: [LiquidityPositionSnapshot!]! @derivedFrom(field: "pair") mints: [Mint!]! @derivedFrom(field: "pair") burns: [Burn!]! @derivedFrom(field: "pair") swaps: [Swap!]! @derivedFrom(field: "pair") }

type User @entity { id: ID! liquidityPositions: [LiquidityPosition!] @derivedFrom(field: "user") usdSwapped: BigDecimal! }

type LiquidityPosition @entity { id: ID! user: User! pair: Pair! liquidityTokenBalance: BigDecimal! }

saved over time for return calculations, gets created and never updated

type LiquidityPositionSnapshot @entity { id: ID! liquidityPosition: LiquidityPosition! timestamp: Int! # saved for fast historical lookups block: Int! # saved for fast historical lookups user: User! # reference to user pair: Pair! # reference to pair token0PriceUSD: BigDecimal! # snapshot of token0 price token1PriceUSD: BigDecimal! # snapshot of token1 price reserve0: BigDecimal! # snapshot of pair token0 reserves reserve1: BigDecimal! # snapshot of pair token1 reserves reserveUSD: BigDecimal! # snapshot of pair reserves in USD liquidityTokenTotalSupply: BigDecimal! # snapshot of pool token supply liquidityTokenBalance: BigDecimal! # snapshot of users pool token balance }

type Transaction @entity { id: ID! # txn hash blockNumber: BigInt! timestamp: BigInt!

This is not the reverse of Mint.transaction; it is only used to

track incomplete mints (similar for burns and swaps)

mints: [Mint]! burns: [Burn]! swaps: [Swap]! }

type Mint @entity {

transaction hash + "-" + index in mints Transaction array

id: ID! transaction: Transaction! timestamp: BigInt! # need this to pull recent txns for specific token or pair pair: Pair!

populated from the primary Transfer event

to: Bytes! liquidity: BigDecimal!

populated from the Mint event

sender: Bytes amount0: BigDecimal amount1: BigDecimal logIndex: BigInt

derived amount based on available prices of tokens

amountUSD: BigDecimal

optional fee fields, if a Transfer event is fired in _mintFee

feeTo: Bytes feeLiquidity: BigDecimal }

type Burn @entity {

transaction hash + "-" + index in mints Transaction array

id: ID! transaction: Transaction! timestamp: BigInt! # need this to pull recent txns for specific token or pair pair: Pair!

populated from the primary Transfer event

liquidity: BigDecimal!

populated from the Burn event

sender: Bytes amount0: BigDecimal amount1: BigDecimal to: Bytes logIndex: BigInt

derived amount based on available prices of tokens

amountUSD: BigDecimal

mark uncomplete in ETH case

needsComplete: Boolean!

optional fee fields, if a Transfer event is fired in _mintFee

feeTo: Bytes feeLiquidity: BigDecimal }

type Swap @entity {

transaction hash + "-" + index in swaps Transaction array

id: ID! transaction: Transaction! timestamp: BigInt! # need this to pull recent txns for specific token or pair pair: Pair!

populated from the Swap event

sender: Bytes! from: Bytes! # the EOA that initiated the txn amount0In: BigDecimal! amount1In: BigDecimal! amount0Out: BigDecimal! amount1Out: BigDecimal! to: Bytes! logIndex: BigInt

derived info

amountUSD: BigDecimal! }

stores for USD calculations

type Bundle @entity { id: ID! ethPrice: BigDecimal! # price of ETH usd }

Data accumulated and condensed into day stats for all of Uniswap

type UniswapDayData @entity { id: ID! # timestamp rounded to current day by dividing by 86400 date: Int!

dailyVolumeETH: BigDecimal! dailyVolumeUSD: BigDecimal! dailyVolumeUntracked: BigDecimal!

totalVolumeETH: BigDecimal! totalLiquidityETH: BigDecimal! totalVolumeUSD: BigDecimal! # Accumulate at each trade, not just calculated off whatever totalVolume is. making it more accurate as it is a live conversion totalLiquidityUSD: BigDecimal!

txCount: BigInt! }

type PairHourData @entity { id: ID! hourStartUnix: Int! # unix timestamp for start of hour pair: Pair!

reserves

reserve0: BigDecimal! reserve1: BigDecimal!

derived liquidity

reserveUSD: BigDecimal!

volume stats

hourlyVolumeToken0: BigDecimal! hourlyVolumeToken1: BigDecimal! hourlyVolumeUSD: BigDecimal! hourlyTxns: BigInt! }

Data accumulated and condensed into day stats for each exchange

type PairDayData @entity { id: ID! date: Int! pairAddress: Bytes! token0: Token! token1: Token!

reserves

reserve0: BigDecimal! reserve1: BigDecimal!

total supply for LP historical returns

totalSupply: BigDecimal!

derived liquidity

reserveUSD: BigDecimal!

volume stats

dailyVolumeToken0: BigDecimal! dailyVolumeToken1: BigDecimal! dailyVolumeUSD: BigDecimal! dailyTxns: BigInt! }

type TokenDayData @entity { id: ID! date: Int! token: Token!

volume stats

dailyVolumeToken: BigDecimal! dailyVolumeETH: BigDecimal! dailyVolumeUSD: BigDecimal! dailyTxns: BigInt!

liquidity stats

totalLiquidityToken: BigDecimal! totalLiquidityETH: BigDecimal! totalLiquidityUSD: BigDecimal!

price stats

priceUSD: BigDecimal! }

gitcoinbot commented 3 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Workers have applied to start work.

These users each claimed they can complete the work by 265 years, 1 month from now. Please review their action plans below:

1) controlcpluscontrolv has applied to start work _(Funders only: approve worker | reject worker)_.

Sorry wasn't sure if you were still responding, will get back to work on the bounty now that I have schema, will get it done!

Learn more on the Gitcoin Issue Details page.

johnmgrimm commented 3 years ago

Hi @daodaoswap are you still looking for someone who can help you with this issue? I would be happy to help. As far as I understand you want to get a Java code that will read data from the Uniswap Subgraph https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2 and then save it to the database e.g. PostgreSQL. Would you like it to be a Spring service that will expose API that will trigger syncing the data for a given timespan on each call or should it be rather a background java service scheduled internally by for example Spring Boot Scheduler?