search

filecoin-project / devgrants

đź‘ź Apply for a Filecoin devgrant. Help build the Filecoin ecosystem!
Other
377 stars 308 forks source link

RFP Application: Block Explorer with FVM / FEVM support #1106

Closed levlite closed 1 year ago

levlite commented 2 years ago

RFP Proposal: Block Explorer with FVM / FEVM support

Name of Project: Laconic IPLD Block Explorer with FVM/FEVM Support

Link to RFP: Block Explorer with FVM / FEVM Support

RFP Category: core-dev app-dev

Proposer: levlite

Do you agree to open source all work you do on behalf of this RFP and dual-license under MIT and APACHE2 licenses?: Yes

Project Description

This project addresses the need for a Block Explorer for Filecoin with FVM and FEVM support. To do this, our team would extend the Lily database, including the schemas and data ingest process, which would require that we write a statediffing process for Filecoin. Statediffing enables the ingestion and indexing of the entire Filecoin state in a performant manner by only considering the subsets of state updated within a given block (see below).

We would then add an intermediate caching and indexing layer underneath a fully functional frontend which supports the views requested in the RFP. This intermediate caching layer will fulfill the necessary conversion of relevant subsets of data from the Lily database, a complete index of the entire Filecoin blockchain, into database models which directly support the desired block explorer views/queries. This would allow us to deploy and host a performant and fast block explorer for Filecoin, including Actor Introspection tooling for both EVM and native FVM, transaction data, and transaction metadata, with support for all address classes. The block explorer would provide for account abstraction as well as high-level network and chain data with distillable detail. The data would be both searchable and indexable.

Data supporting the block explorer at both the base indexing layer (Lily) and the intermediate caching layer will need to be verified. This verification will take two forms, extensive integration testing of the code over representative subsets of chain data, and writing code for checking the Lily and intermediate cache databases for completeness and correctness as a whole. Populating the entire historical data for Filecoin into Lily is expected to take weeks to months and will demand additional hardware. As such, for the immediate purposes of this grant, we will focus on populating new FVM data at the head of the chain in real-time. Afterwards, we can go back and populate the historical data.

Development Roadmap

Milestone 1: Extension of the Lily database.

To use Lily as the backend for the FVM/FEVM compatible Filecoin block explorer it will be necessary to add additional tables/indexes to Lily to support the requested views. Namely, it will be necessary to index portions of the HAMT state. To do so in a reasonable amount of time will require writing a “statediffing” service for Filecoin. Such a service processes the entire, or select segments, of the HAMT into the database by considering only the nodes that are updated within a given block (statediffs). Cumulatively, all the statediffs across the chain history constitute a complete view of the Filecoin state at any given block height.

Additionally, we will internalize an IPLD blockstore into the Lily TimescaleDB (using https://github.com/ipfs/go-ds-sql). This local cache of IPLD blocks will greatly improve performance when traversing the IPLD DAG referenced from the various tables in Lily.

Steps to complete Milestone 1:

  1. Fork Lily
  2. Write statediffing service for Filecoin
  3. Index HAMT leaf nodes in Lily to provide additional actor and account introspection
  4. Internalize IPLD blockstore into the Lily database
  5. Add support for all address classes and account abstractions
  6. Unit and integration testing of these new features

Milestone 2: Intermediate caching layer and front end user interface.

We will provide an intermediate caching and indexing layer on top of the Lily database. Lily provides a base layer index of the complete Filecoin blockchain. The data models indexed in Lily do not directly support the required block explorer views. It will be necessary to perform additional transformations on subsets of the data stored in Lily and caching these new models closer to the block explorer in a dedicated database to improve performance.

We will initiate the project using React without any classical state management tool and leverage React Query to provide asynchronous state management. The block explorer frontend will always have up-to-date managed queries and an automatic refetch mechanism. Queries will be cached and the user will always have a smooth and seamless experience.

Steps to complete Milestone 2:

  1. Write database schema for models that directly support the block explorer views
  2. Write code for reading and transforming data from Lily into the above database models
  3. Write the front end view on top of this secondary cache

Milestone 3: Deploy and host block explorer.

We will make sure that the software is deployed and available on a specified domain name (as well as any extra aliases Filecoin chooses). The proposed domain name is filecoin.explore.gateway.fm. We will provision monitoring and alerting to make sure that the system doesn’t go down and/or exhaust resources.

Steps to complete Milestone 3:

  1. Write the deployment scripts
  2. Provision necessary software into one of the Gateway clusters
  3. Provision Filecoin nodes
  4. Provision the explorer indexer backend
  5. Provision the explorer UI
  6. Add basic monitoring and alerting

Milestone 4: Documentation and education.

Steps to complete Milestone 4:

  1. Extend documentation of Lily to include our changes
  2. Document the statediffing service
  3. Go docs autogenerated from code
  4. Documentation on the block explorer website itself
  5. Document deployment procedures

Milestone 5: Verification of Lily database and block explorer.

Data that currently exists in the Lily database must be verified for completeness and validity before it can be made available via the block explorer.

Steps to complete Milestone 5:

  1. Writing verification code
    1. Verification of the statediffing and Lily code on a representative subset of data
    2. Superficial/first pass validation
      1. Check that records are present in the database for every expected block number
    3. Trailing full validation
      1. Load state for n-1 from the extended Lily database
      2. Load messages for n from the extended Lily database
      3. Apply messages from n on state from n-1 to produce state at n
      4. Confirm that the generated state root at n matches the expected root seen in the Filecoin header at n (outside of Lily)
  2. Verifying the verification code
    1. Integration tests over smaller chain segments
  3. Verifying the production database
    1. Running verification code on the full Lily database

The completeness and validity of the intermediate caching layer and models exposed by the block explorer UI needs to be verified as well. The steps for this will mirror directly the above steps outlined for the Lily database.

Expected Functionality after each Milestone

Milestone 1: Extension of the Lily database.

Lily TimescaleDB extended to support indexing of HAMT state and to include an internalized IPLD blockstore.

Milestone 2: Intermediate caching layer and front end user interface.

Fully functional block explorer frontend supported/accelerated by a TimescaleDB intermediate caching layer.

Milestone 3: Deploy and host block explorer.

Deployed and hosted, full-fledged block explorer supporting the requested Data to Display, with web UI.

Milestone 4: Documentation and education.

Documentation for the Filecoin statediffing service and changes to the Lily database.

Milestone 5: Verification of Lily database and block explorer.

Well-documented verification code and process for the Lily database and block explorer UI models.

Final functionality: Fully populated Lily database and intermediate caching layer with a block explorer that covers the entire history of Filecoin, offering an array of views including Actor Introspection, complete transaction data, metadata, all address classes, support for abstract accounts and delegated abstractions, with searchability and indexability.

Documentation, Education, and Community

Gateway.fm will create a Github repository for the Knowledge base to incorporate all the relevant information on the functionality for the IPLD block explorer with FVM and FEVM support.

Gateway.fm and Vulcanize will promote it through our website and to our contacts in the community (Encode Club) Discord channels, Twitter, etc.

Milestone Summary

Milestone No. Milestone Summary & Staffing Funding Estimated Timeframe Target Completion
1 Extension of Lily, 2 FT Sr Devs $52,800 6 dev weeks 11/29/2022
2 Caching Layer and Frontend, 2 FT Devs, FE and BE $52,800 6 dev weeks 12/13/2022
3 Deploy and Host, 1 FT Dev $35,200 4 dev weeks 12/28/22
4 Documentation, 1 FT Dev $17,600 2 dev weeks 12/28/22
5 Verification, 2 FT Sr Devs $52,800 6 dev weeks 1/10/2023
6 Maintenance and Support for three years $20,000 3 years 1/10/2026

Total Budget Requested

$231,200

Maintenance, Hosting, and Upgrade Plans

The plans for maintenance and upgrades include iterative improvements with plans for native FVM updates next year. Additionally, Vulcanize and Gateway would provide ongoing maintenance, including:

  1. Ongoing support for populating the Lily database in real-time indefinitely into the future
  2. Ongoing support for hosting web UI
  3. Ongoing support for hosting docsite
  4. Ongoing code maintenance
    1. Fixing bugs
    2. Making adjustments
    3. Performance improvements
    4. Adapting to any potential changes in the Filecoin protocol/data structure

Finally, in terms of upgrades, Vulcanize intends to provide for historical state and to track changes as Filecoin/FVM/FEVM development progresses.

Team

Vulcanize Gateway

Contact Info

leah@vulcanize.io rick@vulcanize.io

Team Members

Team Websites

vulcanize.io gateway.fm

Relevant Experience

Vulcanize, Inc. has been immersed with IPLD and IPFS since 2017. Specializing in mechanism design with an emphasis on proof-of-stake systems, Vulcanize advises its clients on an assortment of topics related to the development and deployment of blockchain-based systems. Vulcanize has received grants from Protocol Labs, Interchain Foundation, Uniswap, MolochDAO, and from the Ethereum Foundation for its contribution to EIP-1559.

Gateway.fm is a distributed blockchain infrastructure company. Our mission is to provide enterprise-grade blockchain infrastructure at scale, distributed around the world, in a portable way. We focus on all things infra, sre, and devops: robust and high-performance RPCs, validators at scale, testnets and devnets including all the tooling needs for chains (block explorers, faucets, metrics, etc). Our contributions to the community ranges from writing best practices for ethereum validators, to operations, management and monitoring.

We also put a lot of effort into federating the infrastructure layer both from a physical position to a regulatory position to be as secure as possible.

We built our systems in a very portable way. No cloud provider vendor lock, minimal SaaS dependency: we can run our stack just on bare metal with a linux installation on it. We already support six independent locations across the globe with more incoming, we are in the process of continuing expansion to allocate new locations. As part of the development of the new IPLD block explorer with FVM and FEVM support, Gateway will host, operate, and manage the accessibility to the tool as part of this project.

Team code repositories

https://github.com/cerc-io/go-ethereum/tree/v1.10.25-statediff-v4/statediff#readme https://github.com/cerc-io/ipld-eth-db https://github.com/cerc-io/ipfs-ethdb https://github.com/cerc-io/ipld-eth-server https://github.com/cerc-io/ipld-eth-db-validator https://github.com/cerc-io/ipld-eth-state-snapshot https://github.com/vulcanize/go-codec-dageth https://github.com/vulcanize/ts-dag-eth https://github.com/vulcanize/go-codec-dagcosmos

eshon commented 1 year ago

Closing in lieu of a revised proposal for Filecoin ETL Laconic has shared.