States crawler - Githubissues

Andrei-Dolgolev commented 2 years ago

Existing problems

1: biologist crawler

Currently biologist have 2 main parts bisedes calculation it interacting with QueryAPI and Blockchain.

If we will have state of view method in database we can remove requirements interacting with Blockchain and Multicall contracts.

2: Opensea of locked assets

Currently for be able get understanding if asset is locked they need call method if unicorn is locked.

If we have assets view method state we can get understanding if asset locked or not it can good api for users.

3: Get stats of NFTS

Usual you need write crawler of metadata url and renew current metadata state.

Suggetion of state crawler version 1 (crawl repeat by interval, 1 blockchain, brownie)

Crawling tasks:

Simple task

get Total supply:

{
    "type": "function",
    "name": "totalSupply",
    "outputs": [
        {
            "internalType": "uint256",
            "name": "",
            "type": "uint256",
        }
    ],
    "address": "0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f",
    "inputs": [], # it describe inputs for interface(regular abi generation and encoding and decoding call data)
     "value":[] # it describe inputs for crawler
}

Task have address and all requred data fro generate Contract interface for make call cross Multicall2 and decode output.

Value: Can be:

str - simple 1 value
list - set of values for check(say list of token_ids or addresses)
dict - object of subcall(same structure inside) for another view method, or call as example for all token holders wich can provided by QueryAPI.

Because we have subdependency between input and output of different view methods(see case 1):

Complex task

nested structure:

For repeat logic of task from case 1:

Required chain of calls:

totalSupply -> getDNA -> getUnicornBodyParts

Because getDNA need range of current tokens and getUnicornBodyParts required DNA as input.

{
        "name": "getUnicornBodyParts",
        "address": "0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f",
        "type": "function",
        "inputs": [
            {
                "name": "_dna",
                "type": "uint256",
                "internalType": "uint256",
                "value": {
                    "type": "function",
                    "inputs": [
                        {
                            "internalType": "uint256",
                            "name": "_tokenId",
                            "type": "uint256",
                            "value": {
                                "type": "function",
                                "name": "totalSupply",
                                "outputs": [
                                    {
                                        "internalType": "uint256",
                                        "name": "",
                                        "type": "uint256",
                                    }
                                ],
                                "address": "0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f",
                                "inputs": [],
                            },
                        }
                    ],
                    "name": "getDNA",
                    "outputs": [
                        {"internalType": "uint256", "name": "", "type": "uint256"}
                    ],
                    "address": "0xdC0479CC5BbA033B3e7De9F178607150B3AbCe1f",
                },
            }
        ],
        "outputs": [
            {"internalType": "uint256", "name": "bodyPartId", "type": "uint256"},
            {"internalType": "uint256", "name": "facePartId", "type": "uint256"},
            {"internalType": "uint256", "name": "hornPartId", "type": "uint256"},
            {"internalType": "uint256", "name": "hoovesPartId", "type": "uint256"},
            {"internalType": "uint256", "name": "manePartId", "type": "uint256"},
            {"internalType": "uint256", "name": "tailPartId", "type": "uint256"},
            {"internalType": "uint8", "name": "mythicCount", "type": "uint8"},
        ],
    }

For resolving that nesting we can use next algorithm:

We parse tasks as on picture:

How store

As suggest @zomglings just put it in same labels table.

label_name = "view-state"

label = label_model(
        label=label_name,
        label_data={
            "type": "view",
            "name": call["name"],
            "result": call["result"],
            "inputs": call["inputs"],
            "status": call["status"],
        },
        address=call["contract_address"],
        block_number=call["block_number"],
        transaction_hash=None,
        block_timestamp=call.block_timestamp,
    )

Inputs and outputs decoding

In moonworm crawler we have decoder of transactions wich parse output to dictionary.

We need do same for args of inputs and output parameters with provided abi

Currently multicall contractract return us a tuple.

Sometime abi not have names for output and inputs:

    "inputs": [
        {
            "internalType": "uint256",
            "name": "_tokenId",
            "type": "uint256",
            "value": "af4ac3c0fa304befcc08b293f35d8d95"
        }
    ],
    "level": 1,
    "name": "getDNA",
    "outputs": [
        {
            "internalType": "uint256",
            "name": "",
            "type": "uint256"
        }
    ],
    "stateMutability": "view",
    "type": "function"

but sometime it required for difficult output as on example from task upper:

....
            {"internalType": "uint256", "name": "bodyPartId", "type": "uint256"},
            {"internalType": "uint256", "name": "facePartId", "type": "uint256"},
            {"internalType": "uint256", "name": "hornPartId", "type": "uint256"},
            {"internalType": "uint256", "name": "hoovesPartId", "type": "uint256"},
            {"internalType": "uint256", "name": "manePartId", "type": "uint256"},
            {"internalType": "uint256", "name": "tailPartId", "type": "uint256"},
            {"internalType": "uint8", "name": "mythicCount", "type": "uint8"},
....