rivine explorer dumps all info -> redis server

[despiegk]

these specs are prob double & need to go elsewhere, but need something to organize our thoughts.

address = 
tokens_amount = 
locked_up_till = epoch
multisign = [[address,state],...] # state is bool

address means wallet address

[GlenDC]

Some questions for your @despiegk, please answer them as soon as you can here, either here via telegram or by giving me a call:

• What about the block stakes, we ignore them?
• What is the state in [address,state] of the multisign property?
• When it is a multisign, does that mean that you want the root address to be the multisign wallet address (starting with 0x03)?
• You have a root property locked_up_till, but for a given address, it is possible that you have multiple different locked_up periods, as that is a property per output, so I feel that this property might need to change into a locked_tokens_amount = [epoch->amount]?

No matter the answer on those questions, here follows a low-effort proposal which offers full flexibility.

The "Redis Rivine Explorer" (let's call it rexplorer for the sake of this spec) will be a separate binary, running as a full client within a Rivine network (e.g. the tfchain network).

The binary will have three modules:

• (1) A Gateway module: this one is already created, and can be used directly from the Rivine repo. This module —also used by the Rivine daemon— is used to connect to other peers within the network and receive incoming block diffs;
• (2) A Consensus module: this keeps track of the state of the blockchain at the last known height, applies the rules and ensures all blocks are valid within the context of the agreed upon rules. This module is also to be used directly from the Rivine Repo, as it is used by the Rivine daemon as well;
• (3) A Statistics module: this keeps track of specific statistics;

Only (3) is to be developed to make the rexplorer a reality, meaning we can immediately work on the business logic, without having to make any of the primitives ourselves.

Probably we want to only care about confirmed blocks (meaning they are part of a created block), and ignore the unconfirmed blocks (the ones only in the transaction pool). Note that this doesn't mean the blocks cannot be reverted. This is important to keep in mind, as forks can still happen. As a consequence it means that some address can disappear or have its property change due to this property. I do not foresee any issues with this though, given that spending a token can have the same effect.

The Statistics module uses a redis client, configured using address and database (num) flags. It will store the info for each address directly in that database using the address as the root key.

NOTE: this isn't clear to me, or perhaps I misunderstand your proposed multisign property, but do you also want to reference in wallet addresses any multisig address it is part of?

Given these statistics one can query the redis server for a lot of info. Here are some examples:

• What is the amount of unlocked tokens for a given address?
• What is the total amount of unlocked tokens on the chain (would need to sum all the found addresses, can be quite heavy)
• ...

Many more queries are possible, a lot depends on what stats you want to keep for it.

In order to make queries about global facts (totals for all addresses, block height, ...), we could also keep an object to track such global state. This object could be used to store the current block height, current blockchain time (epoch timestamp of last block).

---

I would like to resolve all questions I have, and you might have first. Get aligned on this, prior to the start of the implementation, as to avoid wasting time on stuff that wasn't asked for.

The proposal is however pretty low-effort, and thus we can get a MVP within a day or 2, from start of implementation.

[GlenDC]

rexplorer (the redis-based explorer binary) is finished (tagged also a first pre-release v0.1.0). You can find it under: https://github.com/threefoldfoundation/rexplorer. There is one open bug https://github.com/threefoldfoundation/rexplorer/issues/1, on testnet it is getting weirdness with the reverting/spending of unexisting (unspent) coin outputs, but on standard network it is already working fine as it is.

Please verify @rkhamis that this binary is working as intended.

[GlenDC]

Added some more global statistics that are tracked, and updated all examples and docs to reflect that.

I also added a proposal at https://github.com/threefoldfoundation/rexplorer/issues/3 as to provide a redis channel API for address/block updates. Please checkout that proposal @despiegk and @rkhamis, and let me know if something like this is desired. It wasn't requested, but I think we require something like this if we want to prevent heavy polling.

[GlenDC]

Released a first pre-release version of rexplorer, v0.1.1. You can find find the tag, pre-compiled binaries and release notes at https://github.com/threefoldfoundation/rexplorer/releases/tag/v0.1.1. It is ready for verification and has everything and more that is asked, AFAIK.

Please verifiy when you can @despiegk and/or @rkhamis.

Next version I aim for is v0.2.0, another pre-release. It would add some more tests. But more importantly I would like it to have it a Redis-channel backed subscribe API, so you can consume the data without having to poll the Redis database at all times. See my proposal at <threefoldfoundation/rexplorer#3> for more information.

Next version of rexplorer will have to wait a bit however, as I have now other more urgent priorities. Please do verify the last release of rexplorer as to see if it fulfils your needs.

[GlenDC]

Had a meeting with @despiegk today. Here is his feedback —as I understood it— summarised:

• make it simpler: even though it is already pretty simple, it is still too technical for non-technical people (see: threefoldfoundation/rexplorer#9)
• less is more: there is still too much public data we store: there is for example no point in showing an entire coin output for the locked outputs, if all is desired is the amount, locked until and the description (taken from arbitrary data usually) (see: threefoldfoundation/rexplorer#9)
• optimise for performance and space, taken into account how Redis works under the hood (see: threefoldfoundation/rexplorer#10)
• give a choice of data format: right now we enforce JSON, but msgpack should be a supported option as well (see: threefoldfoundation/rexplorer#8)
• no need for key namespaces for the different chains and networks (e.g. tfchain:standard:). Instead a different Redis database (slot) should be used for each network (see: threefoldfoundation/rexplorer#11)

All issues are planned in as part of milestone https://github.com/threefoldfoundation/rexplorer/milestone/2, delivered next week.

[GlenDC]

v0.1.2 has been released of rexplorer. Tag, pre-compiled binaries and release notes can be found at https://github.com/threefoldfoundation/rexplorer/releases/tag/v0.1.2. It applied all suggestions of @despiegk —to my best knowledge— from our last meeting.

Please verifiy when you can @despiegk and/or @rkhamis.

[GlenDC]

v0.2.0 has been released of rexplorer. Tag, pre-compiled binaries and release notes can be found at https://github.com/threefoldfoundation/rexplorer/releases/tag/v0.2.0. It applied all suggestions of @despiegk —to my best knowledge— from our last meeting, and some extra stuff on top of it.

Please verifiy when you can @despiegk and/or @rkhamis.

To summary what is to interest of you @despiegk:

• you can now specify (glob) filters using -f <pattern> to define one or multiple filters (one -f flag per pattern), such that all unlocked outputs which have a description that matches one of the given pattern will be tracked in the wallet value as well;
• JSON and MessagePack encodings are now more efficient;
• protobuf is now supported as an encoding format as well, it is slightly more compact than MessagePack but slightly slower as well (MessagePack is quite fast in the current version);
• added an integration test to ensure that Python 3 can decode and use the public data in the Redis DB easily, so you should have no issues using the data, using any of the available encoding formats (JSON, MessagePack, Protobuf).

Please check the release notes for a full overview in the release notes on the release page.

[GlenDC]

By the way @despiegk, I was thinking lately. I think if we add a web server tool to this repo, which serves the redis data in the form of a nice and compact website, it could be very useable for end users as well who have no technical knowledge. Even for power users it can be nice, even though they might spawn their own web server using our provided binary. But for the non technical users we could provide the web service as a public service on one of our domains as we already do for the tfchain web explorers as well.

What do you think?

[despiegk]

good idea, was planned to be part of 3bot

[GlenDC]

I did think it would have already crossed your mind. If you already have a plan for 3bot, than that is great. Have closed and marked the proposal at https://github.com/threefoldfoundation/rexplorer/issues/32 with more details as wontfix in the meanwhile.

[GlenDC]

v0.2.1 has been released of rexplorer. Tag, pre-compiled binaries and release notes can be found at https://github.com/threefoldfoundation/rexplorer/releases/tag/v0.2.1.

The most important thing for the user is that filters now can be updated for an existing redis-db. If the user changes a filter (adding and/or removing filter(s)) they'll get a Y/N question as to warn them that the db will be modified if they answer y. If they want to answer yes automatically they can use the -y (or --yes) flag to answer yes to that question automatically (especially useful for automated environments).