search

web3go-xyz / web3go.xyz

Discovery data value of Web3
Apache License 2.0
11 stars 4 forks source link

Data structure on Substrate #136

Closed bianyunjian closed 8 months ago

bianyunjian commented 10 months ago

Dashboard:

  1. https://analytix.web3go.xyz/layout/dashboardDetail/1f72ec9d-5557-48f9-9d21-2a426b485007
  2. https://analytix.web3go.xyz/layout/dashboardDetail/11f029aa-a521-46f2-9ab4-663c1052295f

now all data comes from excel or static data, just provided for internal checking

Target:

  1. use On-chain data
bianyunjian commented 10 months ago

Manta Atlantic Overview ==> Calamari
https://calamari.subscan.io/ https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fcalamari.systems#/explorer

bianyunjian commented 10 months ago

temp repo: https://github.com/web3go-xyz/manta-datakit-calamari-indexer

extra doc from Manta team: https://hackmd.io/o6aaDgBbQF-jx1BVfEjEAA?view

bianyunjian commented 10 months ago

temp dev server:

Host w3-dev-manta-datakit
    HostName 35.187.244.140
    User ubuntu
    IdentityFile  manta-datakit-private.openssh

manta-datakit-private.zip

bianyunjian commented 10 months ago

Start with:

  1. learn basic raw data: substrate_block_entities,substrate_extrinsic_entities, substrate_event_entities
  2. migrate data to Kafka, Presto, Kudu
  3. parse the data with our own bigdata framework
zhangqixin9527 commented 10 months ago

start developing it for sending the entities to kafka

bianyunjian commented 10 months ago

now data has been sent to Kafka successfully.

zhangqixin9527 commented 10 months ago

I have created tables in Kudu and set up nifi to insert data into tables.

Hope to give me a server to deploy the Calamari Indexer

zhangqixin9527 commented 10 months ago

The program has been deployed successfully, and the data has been recorded in to Kudu tables.

bianyunjian commented 10 months ago

start to fetch blocks after 3M

wangminqi commented 10 months ago

https://www.notion.so/web3go/Manta-Substrate-ecdebfcce16d4c46b8c9bf387b3d981e

zhangqixin9527 commented 10 months ago

try to fix the issue of the calamari-indexer program interruption issue , i found there is throtling on ratelimited endpoint

bianyunjian commented 10 months ago

new archive nodes will solve the problem above.

bianyunjian commented 10 months ago

fixed, now indexer is ready.

zhangqixin9527 commented 10 months ago

The program is well but sending Kafka messages from calamari-indexer fails. I’m attempting to adjust the Kafka producer parameters and will continue monitoring the program.

Currently, the program is sending messages to Kafka, but the processing speed is very slow.