oleiba
commented
4 years ago Great! Just to be a bit more specific, ideally we would want all the relevant fields as we get from an operation when we send a transaction, i.e.:
- hash
- fee
- value
- data
- gasLimit, gasPrice
- source
- destination
- counter
- blockheight, blockhash.
We would want the history to include both unconfirmed and confirmed transactions (distinguished by the unconfirmed txs having default block values). Lastly, this object can be reused also for streaming events for new transactions (https://github.com/ecadlabs/taquito/issues/186).
Innkst
zamrokk
jevonearth
BearCooder
Overview
Taquito will provide an indexer-agnostic library for convenient access to indexed Tezos data. Forming a vertically integrated stack is a Taquito anti-goal.
Problem Statement
Why Indexers are needed?
A Blockchain is a linked list of blocks that contain operations. This data structure is optimized for its purpose as a blockchain, a distributed ledger of operations with strong correctness guarantees.
A linked list of blocks containing operations does not lend itself to querying & reporting of data in the blockchain.
A Blockchain indexer reads data from the blockchain, starting at block zero and progressing to the latest "head" block. At each block, the Indexer extracts the block's "operations," transforms it into a new data structure, and writes that data to a data store (typically RDBMS or a Column-orientated DBMS). Blockchain Indexers typically provide a web API from which users can query transaction data from the blockchain efficiently.
The Tezos indexers each offer their APIs, and each Indexer has offers common features and features unique to that Indexer.
Why does the Tezos Node not provide basic indexing features?
Putting indexing features in the Tezos node would bloat the Tezos Node and is not an attractive proposition.
Prominent Tezos Indexers
Links to the API docs of existing Tezos indexers https://tzstats.com/docs/api/index.html https://api.tzkt.io/ https://github.com/Cryptonomic/Conseil/wiki/REST-API https://tezgraph.com/ / https://gitlab.com/nomadic-labs/tezos-indexer
"TZScan" was the first Tezos Indexer and had considerable market share as well as many dependent projects. The authors of the TZScan Indexer decided to leave the ecosystem. In doing so, they shut down their Indexer, leaving dependent projects scrambling to find new sources of indexed data.
Nomadic's "tezos-indexer" is an ETL daemon that indexes the Tezos blockchain into a Postgresql database. There's an effort named TezGraph to build a GrapQL API on top of this database.
Proposed Solution
Offer a new Taquito package named
@taquito/idxthat offers the most common requirements for indexed data. The idx package is an interface with individual providers for each indexer API. Similar to the Taquito Wallet API (Beacon, TezBridge) or the Signer API (InMemory, Remote, TezBridge)A user of Taquito can use a single query API and choose the backend indexer API that best suits their needs and are most comfortable with it.
Exposed Data
NTD: Incomplete, needs further definition and validation against each Indexer
Implementation Considerations
Should Taquito house each indexer provider within the Taquito git repository?
No. Indexer APIs can change rapidly, and that change is outside the control of the Taquito maintainers. We plan to host indexer plugins that we participate in maintaining as separate git repositories. This will decouple the release cycles of Taquito from the release cycles of Indexer API vendors.
How to handle unsupported data If an upstream indexer does not support a specific data set, an E_NOT_IMPLEMENTED error will be returned to the user. Big Maps is an area where this scenario may arise as not all indexers support big maps yet.
Interface Design
ToDo:
Provider Implementation Considerations
A provider should be instantiated and injected into the @taquito/idx instance. At instantiation, the provider instance may require custom configuration, an API key or a correlation ID for distributed tracing considerations.
Testing
Ongoing testing
Taquito plans to set up a continuous testing harness that runs the indexer queries against all supported Indexer providers. This level of testing aims to discover & detect regressions in the client libraries and the upstream Indexer APIs early.
NTD: Discuss the possibility of collaboration and alerting with indexer vendors. Discuss the labour commitment in maintaining this idea.
Indexer Standardization / Implementation opportunities
By implementing a common, albeit narrow API for multiple vendors, we will be challenged to see that each Indexer provides consistent data facts. We also face risk on data transcoding where the shape or representation from an indexer needs to be coerced into a common data type returned from the query API.
This challenge offers a research and testing opportunity to see that the various indexers represent data accurate to the blockchain's expected representation.
Rejected Ideas
Here we summarise alternative approaches that we have abandoned or rejected.
Choose one Indexer and ship a client for that Indexer
Coupling to one Indexer would be technically more straightforward and expose all data from that source. However, we believe it is a worthy goal not too tightly coupled to upstream projects. Taquito works with the Tezos blockchain from the "bottom-up," and when it can't, in this case, it must rely on indexers. So we are choosing to offer a narrower set of data while maintaining freedom of choice for Taquito users.