[autonity-demo] Blockscout

Klazomenai commented 4 years ago

User story

As Clearmatics, We want to get Blockscout working, so that we can decide what block explorer to focus our efforts in future.

Description

Epirus licences are not financially sustainable across multiple environments, and decisions were made to explore other block explorers. The purpose of this task is to use the autonity-demo chart structure to create a non production ready Helm Chart for demo purposes.

Persistent data not required at this stage

Acceptance criteria

[ ] Use the BlockSout non production Docker image build procedures
[ ] Where possible, use only Kubernetes native stacks

TODO

[ ] Deploy autonity-network to playground cluster
[ ] Deploy pgsql using stable chart to playground cluster
[ ] Build and update docker image for https://hub.docker.com/r/clearmatics/blockscout
[ ] Develop new chart blockscout
- [ ] deployment
- [ ] services
- [ ] secrets for pgsql
- [ ] README
- [ ] Release v0.0.1
[ ] Support by autonity-demo as a dependencies blockscout and pgsql charts.

eastata commented 3 years ago

Issue should be decomposed

Deploy autonity-network to playground cluster
Deploy pgsql using stable chart to playground cluster
Build and update docker image for https://hub.docker.com/r/clearmatics/blockscout
Develop new chart blockscout
- deployment
- services
- secrets for pgsql
- README
- Release v0.0.1
Support by autonity-demo as a dependencies blockscout and pgsql charts.

eastata commented 3 years ago

https://github.com/poanetwork/blockscout-terraform

Klazomenai commented 3 years ago

https://github.com/poanetwork/blockscout-terraform

This scope of this task is for demoing the capabilities of Blockscout and not for a production ready deployment procedures using the poanetwork Terraform recommendations. Production ready environments will configured after review of Blockscout in a local playground.

eastata commented 3 years ago

Current state

# Deploy autonity network

kubectl create namespace ap-test
helm install autonity-network charts-ose.clearmatics.com/autonity-network --namespace ap-test --version 1.8.1 --set http_rpc.api="eth\,web3\,net\,shh\,tendermint\,txpool\,debug",ws_rpc.api="eth\,web3\,net\,network\,debug\,txpool\,tendermint"

# Forward ports
kubectl -n ap-test port-forward svc/validator-0 8545:8545
kubectl -n ap-test port-forward svc/validator-0 8546:8546

# Run PostgreeSQL

docker run -d --name postgres \
  -p 5432:5432 \
  -e POSTGRES_PASSWORD="" \
  -e POSTGRES_USER="postgres" \
  -e POSTGRES_HOST_AUTH_METHOD="trust" \
  postgres:12.4

# Run init DB migration
docker run --rm --network=host \
  -e 'MIX_ENV=prod' \
  -e 'DATABASE_URL=postgresql://postgres:@localhost:5432/explorer?ssl=false' \
  -e 'ETHEREUM_JSONRPC_VARIANT=geth' \
  -e 'ETHEREUM_JSONRPC_HTTP_URL=http://localhost:8545' \
  -e 'ETHEREUM_JSONRPC_WS_URL=ws://localhost:8546' \
  clearmatics/blockscout:v3.3.3-beta /bin/sh -c "echo $$MIX_ENV && mix do ecto.create, ecto.migrate"

# Run Blockscout
docker run --name blockscout --network=host -e 'MIX_ENV=prod' \
  -e 'DATABASE_URL=postgresql://postgres:@localhost:5432/explorer?ssl=false' \
  -e 'ETHEREUM_JSONRPC_VARIANT=geth' \
  -e 'ETHEREUM_JSONRPC_HTTP_URL=http://localhost:8545' \
  -e 'ETHEREUM_JSONRPC_WS_URL=ws://localhost:8546' \
  clearmatics/blockscout:v3.3.3-beta /bin/sh -c "mix phx.server"

##### REMOVE ALL #######
docker rm -f postgres
docker rm -f blockscout

helm -n ap-test uninstall autonity-network
kubectl delete namespace ap-test

### PSQL client
# Check last blocks
docker exec postgres psql -h localhost -U postgres -d explorer -c 'SELECT * FROM blocks ORDER BY number DESC LIMIT 10;'

eastata commented 3 years ago

Example request dump:

{"id":0,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x34",true]},{"id":1,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x33",true]},{"id":2,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x32",true]},{"id":3,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x31",true]},{"id":4,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x30",true]},{"id":5,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2F",true]},{"id":6,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2E",true]},{"id":7,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2D",true]},{"id":8,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2C",true]},{"id":9,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2B",true]}

Error:

Retrying.                                                                                                                                                                                                                          
2020-11-10T11:44:47.076 application=indexer fetcher=block_catchup first_block_number=52 last_block_number=43 [error] ** (FunctionClauseError) no function clause matching in EthereumJSONRPC.Block.entry_to_elixir/1               
    (ethereum_jsonrpc 0.1.0) EthereumJSONRPC.Block.entry_to_elixir({"committedSeals", ["v+prHTVqTqvo9zApX2KuOumPDpQWOLg9EICTfgtWCAQRtWJuTOwOZ+Bp+gNKFTCKWNOAt6W9ahk1stOgZM2uNwA=", "9zjNaDNGmQkVVUy8CcKH3Pbzx4W8KODT9clqs3rw+B59j0R
T3eM/MDS8yen1L77GDUyXoqS2HwI2BGKCVoiVBwA=", "CyvzIrl0H398gS+ra6h6+ASMzNNJJxCHk8zfBg5A4mcoxNGKBtVbKKXpG4JYfhtnZzPjO9LeZNSfS/oOMQ1Z/wA="]})                                                                                          
    (elixir 1.10.3) lib/enum.ex:1320: anonymous fn/4 in Enum.into/3                                                                                                                                                                
    (stdlib 3.12.1) maps.erl:232: :maps.fold_1/3                                                                                                                                                                                   
    (elixir 1.10.3) lib/enum.ex:2127: Enum.into/4                                                                                                                                                                                  
    (elixir 1.10.3) lib/enum.ex:1396: Enum."-map/2-lists^map/1-0-"/2                                                                                                                                                               
    (ethereum_jsonrpc 0.1.0) lib/ethereum_jsonrpc/blocks.ex:44: EthereumJSONRPC.Blocks.from_responses/2                                                                                                                            
    (ethereum_jsonrpc 0.1.0) lib/ethereum_jsonrpc.ex:483: EthereumJSONRPC.fetch_blocks_by_params/3                                                                                                                                 
    (indexer 0.1.0) lib/indexer/block/fetcher.ex:132: Indexer.Block.Fetcher.fetch_and_import_range/2

Manual request by curl:

curl -X POST -H "Content-Type: application/json" --data '[{"id":0,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x34",true]},{"id":1,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x33",true]},{"id":2,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x32",true]},{"id":3,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x31",true]},{"id":4,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x30",true]},{"id":5,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2F",true]},{"id":6,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2E",true]},{"id":7,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2D",true]},{"id":8,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2C",true]},{"id":9,"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x2B",true]}]' http://localhost:8545 |jq .

Response (only first 2 id):

[
  {
    "jsonrpc": "2.0",
    "id": 0,
    "result": {
      "committedSeals": [
        "0vREkue3NhToDP4Q25Lj4hqHvQndye9jObeYqqQOBJQhINn6JxTGLxUPdXt4uTw+L+6nwvivaO42frWT0D25aQA=",
        "rxzbORKf3TAxfQiZCl3pj1GQjSwnav1l3n907ivy/z03iWi3+ZdBZj5H4oaRhwmhLmdB7UF4xzyGMwGGSFGN0QE=",
        "I5ZeKZNxK2eVxhXkgbmPGkyyEMiNvjnLhIsB/Fd6jOk8OFvspkfQCHfcmxv296zhazEdJQ293HB1RxMeUpLYzQA="
      ],
      "committee": [
        {
          "address": "0x358f7f642769993473460757ec3e18aa3644d223",
          "votingPower": 50000
        },
        {
          "address": "0x689dc1e8100a972221cdbf2a58a57f8cea6e9837",
          "votingPower": 50000
        },
        {
          "address": "0x7d2921d95aadc0a0da910d5a82f839037717198a",
          "votingPower": 50000
        },
        {
          "address": "0xb8f2f0a57a1bd203d14fa6eab276462428737445",
          "votingPower": 50000
        }
      ],
      "difficulty": "0x1",
      "extraData": "0xf90176f864d894358f7f642769993473460757ec3e18aa3644d22382c350d894689dc1e8100a972221cdbf2a58a57f8cea6e983782c350d8947d2921d95aadc0a0da910d5a82f839037717198a82c350d894b8f2f0a57a1bd203d14fa6eab27646242873744582c350b8417c91ba6cfea1ba19d7362f65d89ad0c1bddf1f92b2ebca8d4aede18600317efe6e0b64f372b9e3cc5ab0aa7feee78f05909ea18f46e869845d22c8489a6f86a50080f8c9b841d2f44492e7b73614e80cfe10db92e3e21a87bd09ddc9ef6339b798aaa40e04942120d9fa2714c62f150f757b78b93c3e2feea7c2f8af68ee367eb593d03db96900b841af1cdb39129fdd30317d08990a5de98f51908d2c276afd65de7f74ee2bf2ff3d378968b7f99741663e47e286918709a12e6741ed4178c73c8633018648518dd101b84123965e2993712b6795c615e481b98f1a4cb210c88dbe39cb848b01fc577a8ce93c385beca647d00877dc9b1bf6f7ace16b311d250dbddc707547131e5292d8cd00c0",
      "gasLimit": "0x5f5e100",
      "gasUsed": "0x0",
      "hash": "0xcd21bbc1037fdea9165f93fb3f74e8909297c50dc630117199e8fa3b3421ef63",
      "logsBloom": "0x00004000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000010000000000000004000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
      "miner": "0x358f7f642769993473460757ec3e18aa3644d223",
      "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
      "nonce": "0x0000000000000000",
      "number": "0x34",
      "parentHash": "0x809481f5e37fbf4c17abd22adb49219fb23f83f05ee4d3766a10c292fc6545f0",
      "pastCommittedSeals": [],
      "proposerSeal": "fJG6bP6huhnXNi9l2JrQwb3fH5Ky68qNSu3hhgAxfv5uC2TzcrnjzFqwqn/u548FkJ6hj0boaYRdIshImm+GpQA=",
      "receiptsRoot": "0x6c93a86d6e25780929fe4e478432dc0b1a029757b7d0d8f2a62c90a96bd91be8",
      "round": 0,
      "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "size": "0x378",
      "stateRoot": "0xaac91b0a40d3c0a9cec26222c1e974b44d66c01e0972aef169063229756a42ea",
      "timestamp": "0x5faa7611",
      "totalDifficulty": "0x35",
      "transactions": [],
      "transactionsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
      "uncles": []
    }
  },
  {
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
      "committedSeals": [
        "ph2lj3eXP4cf5FYKcbUX2YNnGYIQ8jwHNpQwQb6xpzYA3x3WTp9Or00sg+1AdjXeix7nXAGpvPR+FWvqT5y5bgA=",
        "VCCE/+TZDEbXrGGTnFnTBHakX3uvcdXRI5rbDEvlIbcKa9PDnGXcLpSXF6TFE/XZOtPqzmr0H8zKJM9c0aW+gwE=",
        "bcJfAKsniD6KQbJ7DWtNiVLTMxRfh5Yj5esNZsK3zkxsMU93Jsz3pWXKX/V48fZFtF3FNfzLgXoiNUw5WF0h0wE="
      ],
      "committee": [
        {
          "address": "0x358f7f642769993473460757ec3e18aa3644d223",
          "votingPower": 50000
        },
        {
          "address": "0x689dc1e8100a972221cdbf2a58a57f8cea6e9837",
          "votingPower": 50000
        },
        {
          "address": "0x7d2921d95aadc0a0da910d5a82f839037717198a",
          "votingPower": 50000
        },
        {
          "address": "0xb8f2f0a57a1bd203d14fa6eab276462428737445",
          "votingPower": 50000
        }
      ],
      "difficulty": "0x1",
      "extraData": "0xf90176f864d894358f7f642769993473460757ec3e18aa3644d22382c350d894689dc1e8100a972221cdbf2a58a57f8cea6e983782c350d8947d2921d95aadc0a0da910d5a82f839037717198a82c350d894b8f2f0a57a1bd203d14fa6eab27646242873744582c350b84143be7bb727301f494540063ed3798f3ffb66f797af86c750eb4cf6c6b158f5b10bd1e4b5a94dfab7d7c4ba2ff438330b52335200aa47543eeaca80600b0e511e0180f8c9b841a61da58f77973f871fe4560a71b517d98367198210f23c0736943041beb1a73600df1dd64e9f4eaf4d2c83ed407635de8b1ee75c01a9bcf47e156bea4f9cb96e00b841542084ffe4d90c46d7ac61939c59d30476a45f7baf71d5d1239adb0c4be521b70a6bd3c39c65dc2e949717a4c513f5d93ad3eace6af41fccca24cf5cd1a5be8301b8416dc25f00ab27883e8a41b27b0d6b4d8952d333145f879623e5eb0d66c2b7ce4c6c314f7726ccf7a565ca5ff578f1f645b45dc535fccb817a22354c39585d21d301c0",
      "gasLimit": "0x5f5e100",
      "gasUsed": "0x0",
      "hash": "0x809481f5e37fbf4c17abd22adb49219fb23f83f05ee4d3766a10c292fc6545f0",
      "logsBloom": "0x00004000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000010000000000000004000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
      "miner": "0xb8f2f0a57a1bd203d14fa6eab276462428737445",
      "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
      "nonce": "0x0000000000000000",
      "number": "0x33",
      "parentHash": "0x6d4f6df36ca6604e3b71921acfcae5b040c88b699a3312be38a8194c4d44d1b1",
      "pastCommittedSeals": [],
      "proposerSeal": "Q757tycwH0lFQAY+03mPP/tm95evhsdQ60z2xrFY9bEL0eS1qU36t9fEui/0ODMLUjNSAKpHVD7qyoBgCw5RHgE=",
      "receiptsRoot": "0x6c93a86d6e25780929fe4e478432dc0b1a029757b7d0d8f2a62c90a96bd91be8",
      "round": 0,
      "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "size": "0x378",
      "stateRoot": "0xaac91b0a40d3c0a9cec26222c1e974b44d66c01e0972aef169063229756a42ea",
      "timestamp": "0x5faa7610",
      "totalDifficulty": "0x34",
      "transactions": [],
      "transactionsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
      "uncles": []
    }
  }
]

Block structure in Blockscout sources
Elexir lib line

eastata commented 3 years ago

Was tested with BLOCK_TRANSFORMER=base and clique. The same error

Klazomenai commented 3 years ago

Planning update, @aiman to follow up on next decision.

aiman commented 3 years ago

From the error message, it appears that Blockscout is unable to parse the Autonity block structure. It failed at committedSeals, but will likely fail on other ones that are unique to Autonity.

Autonity block structure: https://github.com/clearmatics/autonity/blob/c1348ede267192e904aa90104a8be5a0021c9690/core/types/block.go#L101

eastata commented 3 years ago

Test deployment available here: http://blockscout.eastata.com

Klazomenai commented 3 years ago

@aiman Is Blockscout still to come into our demo Helm charts?

clearmatics / charts-ose-helm3