Substreams Uniswap v3 Data Validation

[x] Lister les data discrepancies qu'on connaît. (Labelled as DD)
[x] Les documenter et expliquer les différences (links to source code au besoin).
[x] S'il y a des data discrepancies qu'on ne connaissait pas, il faut digger pour soit les fixer ou les expliquer.
[ ] Redémarrer le process et le timer.
[ ] Spot-check on some of the latest blocks

Comparing data was done on pool 0x8ad599c3a0ff1de082011efddc58f1908eb6e6d8 at block number: 12375000. Good confidence level, as it is run 6000 blocks after Uniswap-V3 smart contract created.

Bundle

[x] id
[x] ethPriceUSD

Factory

Difference on multiple values SG<>SS but notably SG has totalValueLockedETH set and SS does not.

Command to mimic:

cbwasm && substreams gui substreams.yaml map_factory_entities -e mainnet.eth.streamingfast.io:443 -s 12370680 -t 12370681 --debug-modules-output store_totals,store_native_total_value_locked,map_event_amounts,map_extract_data_types,map_factory_entites

[x] id
[x] poolCount
[x] txCount
[ ] totalVolumeUSD - Affected by bundle.ethPriceUSD
[ ] totalVolumeETH - Affected by bundle.ethPriceUSD
[ ] totalFeesUSD - Affected by bundle.ethPriceUSD
[ ] totalFeesETH - Affected by bundle.ethPriceUSD
[ ] untrackedVolumeUSD - Affected by bundle.ethPriceUSD
[ ] totalValueLockedUSD - Affected by bundle.ethPriceUSD
[ ] totalValueLockedETH - Affected by bundle.ethPriceUSD
[x] totalValueLockedUSDUntracked
[x] totalValueLockedETHUntracked
[x] owner

TODO: A couple of things have changed in the SS notably the store_native_total_value_locked and need to verify that we didn't break anything.

Pool:

[x] id
[x] collectedFeesToken0
[x] collectedFeesToken1
[x] collectedFeesUSD
[x] createdAtBlockNumber
[x] createdAtTimestamp
[x] feeGrowthGlobal0X128 - eth_call should be done on each swap
[x] feeGrowthGlobal1X128 - eth_call should be done on each swap
[x] feeTier
[ ] feesUSD - Affected by bundle.ethPriceUSD
[x] liquidity
[x] liquidityProviderCount
[x] observationIndex
[x] sqrtPrice - sqrtPrice
[x] tick
[x] token0Price
[x] token1Price
[ ] totalValueLockedETH - Affected by bundle.ethPriceUSD
[x] totalValueLockedToken0
[x] totalValueLockedToken1
[ ] totalValueLockedUSD - Affected by bundle.ethPriceUSD
[x] totalValueLockedUSDUntracked
[x] txCount
[ ] untrackedVolumeUSD - Affected by bundle.ethPriceUSD
[x] volumeToken0
[x] volumeToken1
[ ] volumeUSD - Affected by bundle.ethPriceUSD

Token

[x] id
[x] symbol
[x] name
[x] decimals
[x] totalSupply The subgraph contains a bug where they do a BigInt.fromI32(...) with a totalSupply bigger than 2^32 and this causes the SG and the SS to have different values.
[x] volume

After thorough analysis with Alex: SG Query

Shows that the pool.token1.volume is set to 1.443317582511051495 which is a faulty data.

The SS will return 3.3398629999... and cross-referencing with etherscan we find that our data is correct.

[ ] volumeUSD - Affected by bundle.ethPriceUSD

Big difference between the SG and the SS. Check this graphql query here at block 12370680 we get "volumeUSD": "0" however we will get 549.543106851100345873695251819078934899491048634255496718736822003847919404506683349609375 when running the SS. This is normal, because the computation of volumeUSD is based on the bundle.ethPriceUSD which is set to 0 in the SG but is 3306.0105708... for the SS.

The value of bundle.ethPriceUSD is wrong in the SG because it will set it's value on 2 events: Initialize (pool) and Swap. As seen, there is an issue in the SG when there is a PoolCreated then Initialize in the same transaction (or maybe not just the same transaction but simply the Initialize is wrongfully handled). The ethPriceUSD is set by checking the USDC<>WETH pool and it was created and initialized at block 12370624 but the first swap was only registered at block 12371376. Run below query to prove it.

bundle(id: "1", block: {number: $blockNumber}) {
    id
    ethPriceUSD
  }

This means that any entity change field between 12370624 and 12371376 which needs the ethPriceUSD will be different between the SG and the SS.

[ ] untrackedVolumeUSD - Affected by bundle.ethPriceUSD
[ ] feesUSD - Affected by bundle.ethPriceUSD
[x] txCount
[x] poolCount
[x] totalValueLocked
[ ] totalValueLockedUSD - Affected by bundle.ethPriceUSD
[ ] totalValueLockedUSDUntracked
[ ] derivedETH

Again in the same wave length as the point mentioned above with anything being affected by bundle.ethPriceUSD, we have the same issue here.

derivedETH is calculated at 2 events: Initialized and Swap Because the handler handleInitialized in the subgraph contains errors, we wont have good values up until the first Swap of any token. So if you have 10 Mints before any Swap the value of anything that uses the derivedETH value will be faulty.

[x] whitelistPools

Tick

[x] id
[x] poolAddress
[x] tickIdx
[x] pool
[x] liquidityGross
[x] liquidityNet
[x] price0
[x] price1
[x] volumeToken0
[x] volumeToken1
[x] volumeUSD
[x] untrackedVolumeUSD
[x] feesUSD
[x] collectedFeesToken0
[x] collectedFeesToken1
[x] collectedFeesUSD
[x] createdAtTimestamp
[x] createdAtBlockNumber
[x] liquidityProviderCount
[x] feeGrowthOutside0X128
[x] feeGrowthOutside1X128

Position

[x] id
[ ] owner
[x] pool
[x] token0
[x] token1
[x] tickLower
[x] tickUpper
[x] liquidity
[x] depositedToken0
[x] depositedToken1
[x] withdrawnToken0
[x] withdrawnToken1
[x] collectedFeesToken0
[x] collectedFeesToken1
[x] transaction
[ ] feeGrowthInside0LastX128 - Data discrepancy here
[ ] feeGrowthInside1LastX128 - Data discrepancy here Interesting difference here: Running this graphql query

They give you this:

"feeGrowthInside0LastX128": "771019261802862702426015195378793",
"feeGrowthInside1LastX128": "4641159448963217241996835077346319"

And we get 0 for both fields.

Interestingly, going here and going to Contract and running positions with 0x24 (token id value of 36) and we also get 0 for both fields.

There is a bug here. the subgraph with to a getTransaction on any type of position type and if the transaction does not exist, it will create it.

In our case, we can't do that, so, for example, in a transaction we will have 2 positions which will change the same position. This is a problem when let's say you have a Transfer and then a IncreaseLiquidity because both will emit create entity changes and the last one will be persisted. We have to code a "merge" positions where we take all positions in a transaction of the same id and merge them together. There are data points which are exclusive toe each position so merge won't cause an issue.

PositionSnapshot

[x] id
[x] owner
[x] pool
[x] position
[x] blockNumber
[x] timestamp
[x] liquidity
[x] depositedToken0
[x] depositedToken1
[x] withdrawnToken0
[x] withdrawnToken1
[x] collectedFeesToken0
[x] collectedFeesToken1
[x] transaction
[ ] feeGrowthInside0LastX128
[ ] feeGrowthInside1LastX128

Transaction

[x] id
[x] blockNumber
[x] timestamp
[ ] gasUsed - Not the same
[x] gasPrice
[x] mints - derivedFrom
[x] burns - derivedFrom
[x] swaps - derivedFrom
[x] flashed - derivedFrom
[x] collects - derivedFrom

Mint

[x] id
[x] transaction
[x] timestamp
[x] pool
[x] token0
[x] token1
[x] owner
[x] sender
[x] origin
[x] amount
[x] amount0
[x] amount1
[ ] amountUSD - Data Discrepancy here
At block 12374976 we have this transaction which is compromised of token0 (0x6b175474e89094c44da98b954eedeac495271d0f) which is a stable coin.

Adhering to what the subgraph code does:

[...]
if (STABLE_COINS.includes(token.id)) {
    priceSoFar = safeDiv(ONE_BD, bundle.ethPriceUSD)
  } else {
[...]

Subgraph query The value returned and set for token0.derivedETH should've been: 1/3343.219561240329933856038921064461 = 0.00029911287

amount0 = 641.715485603220556627 = 641.72
amount1 = 220.622808166391590159 = 220.62
bundleEthPriceUsd = 3343.219561240329933856038921064461 = 3,343.22
token0DerivedEth = 1/bundleEthPriceUsd = 0
token1DerivedEth = 0 = 0

amountUSD = amount0 * (token0DerivedEth * bundleEthPriceUsd) + (amount1 * token1DerivedEth) = 641.72

The subgraph unfortunately will return a 0 for amountUSD.

[x] tickLower
[x] tickUpper
[x] logIndex

Burn

[x] id
[x] transaction
[x] pool
[x] token0
[x] token1
[x] timestamp
[x] owner
[x] origin
[x] amount
[x] amount0
[x] amount1
[x] amountUSD
[x] tickLower
[x] tickUpper
[x] logIndex

Swap

[x] id
[x] transaction
[x] timestamp
[x] pool
[x] token0
[x] token1
[x] sender
[x] recipient
[x] origin
[x] amount0
[x] amount1
[x] amountUSD
[x] sqrtPriceX96
[x] tick
[x] logIndex

streamingfast / substreams-uniswap-v3