Block explorer requirements

[jarindr]

To build a block explorer ( or an app ) on the network, there are some ongoing questions about how anyone should get the data to display. Many aggregated data is not possible to get from the watcher endpoint now but the explorer directly connects to the watcher db replica instance. some example are:

• get blocks ( watcher endpoint missing )
• get tx count by address
• get tx count by block
• get tx count in the last 24 hours ( or from any given point )
• get block counts ( from any given point )
• average block time ( from any given point )

there are also other things that need to get but is not available on watcher db like

• erc20 token name, symbol, decimal

some solutions are:

• add all the missing part in the watcher information api, explorer talks to it
• explorer has its own db and infrastructure, parsed all the blocks from childchain/watcher db and create its own setup. This is the traditional way of building app / explorer on ETH.
• both, some in watcher some in its own db.

now the explorer talks to both watcher api and watcher db to query the data that are missing which is quite a mess. Now also it needs detail about erc20 which need its own db to cache all the information.

Those pieces of information are stats of the network that open to discuss whether we should put it in the watcher or not.

the data is not only for the explorer, but some stats are also good to have if you want to build an application on top of the network.

[kartsims]

omgpool.org dev here, watching this issue as we intend to use the watcher API to provide additional stats of the network (complementary data to the block explorer's)

[jrhite]

There are still ongoing discussions, but my understanding is that over time we will want to migrate all the informational queries out of watcher and leave watcher as a security only service.

We are building another query service that can also talk to the same DB as Watcher and will provide query endpoints.

I also recommend we add the ERC 20 tokens in a new table, but same DB that Watcher uses.

On Wed, Sep 25, 2019, 14:53 Simon Tarchichi notifications@github.com wrote:

omgpool.org dev here, watching this issue as we intend to use the watcher API to provide additional stats of the network (complementary data to the block explorer's)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/omisego/elixir-omg/issues/1001?email_source=notifications&email_token=AAIJBTF6XKRUMIA7ZCNQR6TQLMKGBA5CNFSM4I2H4UWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7Q6ZAA#issuecomment-534899840, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIJBTAFL5CS6MEE7SW44R3QLMKGBANCNFSM4I2H4UWA .

[pdobacz]

The pinch of historical context here is:

• we intended to have the watcher support the block explorers needs for as long as they don't get too block explorer-specific. I.e. we use watcher's API temporarily to be BE's backend
• we anticipated the split of requirements of block explorer vs app towards the Watcher API

This of course didn't play out ideally, since the block explorer talked to the DB replica and there is close to no "official" support for BE in the Watcher API :joy:, as it seems.

I think it boils down to this part:

the data is not only for the explorer, but some stats are also good to have if you want to build an application on top of the network.

• if so, we should add to the Watcher
• if that's too BE-specific (my vote, but very far from certain), then I'd go with: "explorer has its own db and infrastructure"

Let's also remember that the Watcher may need to do some data pruning/filtering, which would not be relevant for BE.

Last bit is this:

parsed all the blocks from childchain/watcher

Whether child chain or watcher is chosen to feed the BE backend with blocks is an interesting choice:

• child chain - block.get exists already, no work needed there. But it will potentially serve you invalidated blocks
• watcher - block.get (a child chain-compatible version) would have to be added, for the watcher to serve as a proxy to another block consumer (BE). C.f. #1000 too.

Tagging @Pongch for some wider product perspective :)

[unnawut]

Since we are about to have a much clearer boundary between security-critical and informational API, we can iterate faster even within elixir-omg. We can revisit this issue later in time when info API gets bigger.

See the boundary work in #1123, and the main discussion points in #1065.