paradigmxyz / cryo

cryo is the easiest way to extract blockchain data to parquet, csv, json, or python dataframes
Apache License 2.0
1.18k stars 106 forks source link

storage_diffs incorrectly extracted #86

Closed libevm closed 1 year ago

libevm commented 1 year ago

cryo 0.2.0-136-ga4747a4

EDIT

Seems like cryo is extracting the reads?

The data extracted from cryo corresponds to openchain's sload (called via staticcall) - https://openchain.xyz/trace/ethereum/0xea2317f0c0fc06e3cf2a52191c7c5c711e33b0e1ca773f9030867876ea3a4047

image

┌────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┐
│ address                                    ┆ slot                                                               ┆ from_value                                                         ┆ to_value                                                           │
│ ---                                        ┆ ---                                                                ┆ ---                                                                ┆ ---                                                                │
│ str                                        ┆ str                                                                ┆ str                                                                ┆ str                                                                │
╞════════════════════════════════════════════╪════════════════════════════════════════════════════════════════════╪════════════════════════════════════════════════════════════════════╪════════════════════════════════════════════════════════════════════╡
│ 0x6ca298d2983ab03aa1da7679389d955a4efee15c ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0x000000000100010000fcef9900000000000000000002dbf2267afee73ea27bd3 │
│ 0x9cb91e5451d29c84b51ffd40df0b724b639bf841 ┆ 0x087a86b54527aac6fd2b30b56c881d8c77fed05e314a87db1723f16019e4333e ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0xffffffffffffff98ac575d2410fae284000000000000006753a8a2dbef051d7c │
│ 0x9cb91e5451d29c84b51ffd40df0b724b639bf841 ┆ 0x087a86b54527aac6fd2b30b56c881d8c77fed05e314a87db1723f16019e4333f ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0x000000000000000000000000000000001ed3ba8296b3b4b73479051e4c58a084 │
│ 0x9cb91e5451d29c84b51ffd40df0b724b639bf841 ┆ 0x087a86b54527aac6fd2b30b56c881d8c77fed05e314a87db1723f16019e43340 ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0x0000000000000000000000000000000000031b4878f7a5146b979edd757f4c9a │
└────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┘

Platform Linux + Windows (WSL)

Description I'm trying to classify mev bundles and am doing it by finding out the relationship between the slots

In particular this tx: https://etherscan.io/tx/0xea2317f0c0fc06e3cf2a52191c7c5c711e33b0e1ca773f9030867876ea3a4047

I tried this code:

import polars as pl
import cryo
import binascii

pl.Config.set_fmt_str_lengths(100)

def bytes_to_str(x: dict | bytes | list[bytes]) -> str | list[str] | dict:
    if isinstance(x, dict):
        new_dict = {}
        for k, v in x.items():
            new_dict[bytes_to_str(k)] = bytes_to_str(v)
        return new_dict
    if isinstance(x, list):
        return [bytes_to_str(i) for i in x]
    if isinstance(x, bytes):
        return "0x" + binascii.hexlify(x).decode().lower()
    return x

tx_hash = bytes.fromhex('0xea2317f0c0fc06e3cf2a52191c7c5c711e33b0e1ca773f9030867876ea3a4047'[2:])

storage_diff = cryo.collect(
    "storage_diffs",
    blocks=["17719287"],
    output_format="polars",
    rpc="http://localhost:8545",
)
storage_diff = storage_diff.filter(
    pl.col('transaction_hash').eq(tx_hash)
)

print('---- Storage diff found ----')
storage_diff = storage_diff.with_columns(
    pl.col('transaction_hash').map_elements(bytes_to_str),
    pl.col('address').map_elements(bytes_to_str),
    pl.col('slot').map_elements(bytes_to_str),
    pl.col('from_value').map_elements(bytes_to_str),
    pl.col('to_value').map_elements(bytes_to_str),
)
print(storage_diff)

I expected to see no storage slots changing, as the transaction above fails

Instead, this happened:

> python storage_weird.py
---- Storage diff found ----
shape: (4, 8)
┌──────────────┬───────────────────┬────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┬──────────┐
│ block_number ┆ transaction_index ┆ transaction_hash                                                   ┆ address                                    ┆ slot                                                               ┆ from_value                                                         ┆ to_value                                                           ┆ chain_id │
│ ---          ┆ ---               ┆ ---                                                                ┆ ---                                        ┆ ---                                                                ┆ ---                                                                ┆ ---                                                                ┆ ---      │
│ u32          ┆ u32               ┆ str                                                                ┆ str                                        ┆ str                                                                ┆ str                                                                ┆ str                                                                ┆ u64      │
╞══════════════╪═══════════════════╪════════════════════════════════════════════════════════════════════╪════════════════════════════════════════════╪════════════════════════════════════════════════════════════════════╪════════════════════════════════════════════════════════════════════╪════════════════════════════════════════════════════════════════════╪══════════╡
│ 17719287     ┆ 274               ┆ 0xea2317f0c0fc06e3cf2a52191c7c5c711e33b0e1ca773f9030867876ea3a4047 ┆ 0x6ca298d2983ab03aa1da7679389d955a4efee15c ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0x000000000100010000fcef9900000000000000000002dbf2267afee73ea27bd3 ┆ 1        │
│ 17719287     ┆ 274               ┆ 0xea2317f0c0fc06e3cf2a52191c7c5c711e33b0e1ca773f9030867876ea3a4047 ┆ 0x9cb91e5451d29c84b51ffd40df0b724b639bf841 ┆ 0x087a86b54527aac6fd2b30b56c881d8c77fed05e314a87db1723f16019e4333e ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0xffffffffffffff98ac575d2410fae284000000000000006753a8a2dbef051d7c ┆ 1        │
│ 17719287     ┆ 274               ┆ 0xea2317f0c0fc06e3cf2a52191c7c5c711e33b0e1ca773f9030867876ea3a4047 ┆ 0x9cb91e5451d29c84b51ffd40df0b724b639bf841 ┆ 0x087a86b54527aac6fd2b30b56c881d8c77fed05e314a87db1723f16019e4333f ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0x000000000000000000000000000000001ed3ba8296b3b4b73479051e4c58a084 ┆ 1        │
│ 17719287     ┆ 274               ┆ 0xea2317f0c0fc06e3cf2a52191c7c5c711e33b0e1ca773f9030867876ea3a4047 ┆ 0x9cb91e5451d29c84b51ffd40df0b724b639bf841 ┆ 0x087a86b54527aac6fd2b30b56c881d8c77fed05e314a87db1723f16019e43340 ┆ 0x0000000000000000000000000000000000000000000000000000000000000000 ┆ 0x0000000000000000000000000000000000031b4878f7a5146b979edd757f4c9a ┆ 1        │
└──────────────┴───────────────────┴────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┴──────────┘

Querying an archival node via trace_replayTransaction tells me that there is no state changes, so my assumption is the bug is probably how cryo is extracting it?

curl https://docs-demo.quiknode.pro/ \
  -X POST \
  -H "Content-Type: application/json" \
  --data '{"method":"trace_replayTransaction","params":["0xea2317f0c0fc06e3cf2a52191c7c5c711e33b0e1ca773f9030867876ea3a4047",["trace"]],"id":1,"jsonrpc":"2.0"}' | jq
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "output": "0x08c379a0000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000023a28000000000000000000000000000000000000000000000000000000000000",
    "stateDiff": null,
    "trace": [
      {
        "action": {
          "from": "0x0124d0fa0dfb1430dfcff16ec6a96945e7a0bc10",
          "callType": "call",
          "gas": "0x11db68",
          "input": "0xca8bd1f903140100271027109cb91e5451d29c84b51ffd40df0b724b639bf841c02aaa39b223fe8d0a0e5c4f27ead9083c756cc21f000127102710dbbacd44be67ba035925953ff2a8d4430831a8e16e2a43be0b1d33b726f0ca3b8de60b3482b8b05014010001f427106ca298d2983ab03aa1da7679389d955a4efee15cdac17f958d2ee523a2206206994597c13d831ec7074d4fd49e75d8c8087eeb3cbfaf09c80000ab00ab00ab03ab03ab03abdbbacd44be67ba035925953ff2a8d4430831a8e14ff2a47240a8edb3502239da000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000002800000000000000000000000000000000000000000000000000000000e17e78b59000000000000000000000000dac17f958d2ee523a2206206994597c13d831ec70000000000000000000000006e2a43be0b1d33b726f0ca3b8de60b3482b8b0500000000000000000000000001729f93e3c3c74b503b8130516984ced70bf47d90000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000116f7f07d57000000000000000000000000000000000000000000016a752b57fa7c6ced3900000000a4000000a4000000a4000000a400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000a4bf15fcd8000000000000000000000000303389f541ff2d620e42832f180a08e767b28e10000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000242cc2878d0064bf989e000000000000001729f93e3c3c74b503b8130516984ced70bf47d9000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000410290c1c2682ccff84d0ce768adee18b507b8c68b473219a5421f19260a8eb56b50b8831157803802749060615f2c68d22caa2b35efbddb90abafb5d40c72c8701c0000000000000000000000000000000000000000000000000000000000000000000000",
          "to": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "value": "0x0"
        },
        "error": "Reverted",
        "result": {
          "gasUsed": "0x13a59",
          "output": "0x08c379a0000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000023a28000000000000000000000000000000000000000000000000000000000000"
        },
        "subtraces": 3,
        "traceAddress": [],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "delegatecall",
          "gas": "0x7a120",
          "input": "0x6526f12f00000000000000000000000000000000000000000000000000000000000000140000000000000000000000009cb91e5451d29c84b51ffd40df0b724b639bf8410000000000000000000000000000000000000000000000007eeb3cbfaf09c800000000000000000000000000000000000000000000000000000000000000271000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000",
          "to": "0x4ed2b8c9ea349b6fc558f5e8ebca1a53eddcdd9c",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x95a7",
          "output": "0x000000000000000000000000000000000000000000000534e71b3c5106107010"
        },
        "subtraces": 7,
        "traceAddress": [
          0
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x776b8",
          "input": "0x1a686502",
          "to": "0x9cb91e5451d29c84b51ffd40df0b724b639bf841",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x97c",
          "output": "0x00000000000000000000000000000000000000000000074147c08a6f60cd4254"
        },
        "subtraces": 0,
        "traceAddress": [
          0,
          0
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x76bcd",
          "input": "0xd0c93a7c",
          "to": "0x9cb91e5451d29c84b51ffd40df0b724b639bf841",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x117",
          "output": "0x00000000000000000000000000000000000000000000000000000000000000c8"
        },
        "subtraces": 0,
        "traceAddress": [
          0,
          1
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x76920",
          "input": "0xddca3f43",
          "to": "0x9cb91e5451d29c84b51ffd40df0b724b639bf841",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0xfb",
          "output": "0x0000000000000000000000000000000000000000000000000000000000002710"
        },
        "subtraces": 0,
        "traceAddress": [
          0,
          2
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x7668d",
          "input": "0x3850c7bd",
          "to": "0x9cb91e5451d29c84b51ffd40df0b724b639bf841",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0xa88",
          "output": "0x000000000000000000000000000000000000000004e0f04ad2cf8cae0937a699fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffeca9500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001"
        },
        "subtraces": 0,
        "traceAddress": [
          0,
          3
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x75692",
          "input": "0x5339c296fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe",
          "to": "0x9cb91e5451d29c84b51ffd40df0b724b639bf841",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x9dd",
          "output": "0x000000000000000000000080205c4a943ceffe9bca63e9000510000000000000"
        },
        "subtraces": 0,
        "traceAddress": [
          0,
          4
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x7358e",
          "input": "0xf30dba93fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffecb68",
          "to": "0x9cb91e5451d29c84b51ffd40df0b724b639bf841",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x23ef",
          "output": "0x00000000000000000000000000000000000000000000006753a8a2dbef051d7cffffffffffffffffffffffffffffffffffffffffffffff98ac575d2410fae284000000000000000000000000000000001ed3ba8296b3b4b73479051e4c58a0840000000000000000000000000000000000031b4878f7a5146b979edd757f4c9affffffffffffffffffffffffffffffffffffffffffffffffffffffffffe327440000000000000000000000000000000000000000000000001bdaef2594b365e000000000000000000000000000000000000000000000000000000000000000180000000000000000000000000000000000000000000000000000000000000001"
        },
        "subtraces": 0,
        "traceAddress": [
          0,
          5
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x70b81",
          "input": "0x5339c296fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe",
          "to": "0x9cb91e5451d29c84b51ffd40df0b724b639bf841",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x20d",
          "output": "0x000000000000000000000080205c4a943ceffe9bca63e9000510000000000000"
        },
        "subtraces": 0,
        "traceAddress": [
          0,
          6
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x10cb9d",
          "input": "0x7e54f092dbbacd44be67ba035925953ff2a8d4430831a8e14ff2a47240a8edb3502239da",
          "to": "0x1111111254eeb25477b68fb85ed929f73a960582",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x9c0",
          "output": "0x000000000000000000000000000000000000000000000000000001100904c621"
        },
        "subtraces": 0,
        "traceAddress": [
          1
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "delegatecall",
          "gas": "0x7a120",
          "input": "0x6526f12f00000000000000000000000000000000000000000000000000000000000000140000000000000000000000006ca298d2983ab03aa1da7679389d955a4efee15c0000000000000000000000000000000000000000000000000000000401e1242500000000000000000000000000000000000000000000000000000000000001f400000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000",
          "to": "0x4ed2b8c9ea349b6fc558f5e8ebca1a53eddcdd9c",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x585e",
          "output": "0x0000000000000000000000000000000000000000000000007d6401c08f783598"
        },
        "subtraces": 5,
        "traceAddress": [
          2
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x776b8",
          "input": "0x1a686502",
          "to": "0x6ca298d2983ab03aa1da7679389d955a4efee15c",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x97c",
          "output": "0x000000000000000000000000000000000000000000000000117acdc9f6ebf2ac"
        },
        "subtraces": 0,
        "traceAddress": [
          2,
          0
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x76bcd",
          "input": "0xd0c93a7c",
          "to": "0x6ca298d2983ab03aa1da7679389d955a4efee15c",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x117",
          "output": "0x000000000000000000000000000000000000000000000000000000000000000a"
        },
        "subtraces": 0,
        "traceAddress": [
          2,
          1
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x76920",
          "input": "0xddca3f43",
          "to": "0x6ca298d2983ab03aa1da7679389d955a4efee15c",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0xfb",
          "output": "0x00000000000000000000000000000000000000000000000000000000000001f4"
        },
        "subtraces": 0,
        "traceAddress": [
          2,
          2
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x7668d",
          "input": "0x3850c7bd",
          "to": "0x6ca298d2983ab03aa1da7679389d955a4efee15c",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x12ff",
          "output": "0x00000000000000000000000000000000000000000002dbf2267afee73ea27bd3fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffcef99000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000d480d480000000000000000000000000000000000000000000000000000000000000001"
        },
        "subtraces": 0,
        "traceAddress": [
          2,
          3
        ],
        "type": "call"
      },
      {
        "action": {
          "from": "0x9db7378614d8d9d7149c4ee4763f88c38f9b1517",
          "callType": "staticcall",
          "gas": "0x74e3d",
          "input": "0x5339c296ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffb1",
          "to": "0x6ca298d2983ab03aa1da7679389d955a4efee15c",
          "value": "0x0"
        },
        "result": {
          "gasUsed": "0x9c7",
          "output": "0x00fc2100104080ebd08123c9c7fddcebdd4feb2a61b602145e2c840060400002"
        },
        "subtraces": 0,
        "traceAddress": [
          2,
          4
        ],
        "type": "call"
      }
    ],
    "vmTrace": null
  }
}
sslivkoff commented 1 year ago

strange...when I run the same command, I do no see any of those rows being collected

what happens when you have cryo use https://docs-demo.quiknode.pro/ instead of http://localhost:8545? might be a difference in how the endpoints are responding?

sslivkoff commented 1 year ago

also, what kind of endpoint is your localhost endpoint?

sslivkoff commented 1 year ago

spoke with libevm, it as an issue with their node endpoint