Add support for BSQ into the mempool.space explorer, replace our current BSQ Explorers with self-hosted Mempool Explorers

wiz commented 4 years ago

_This is a Bisq Network project. Please familiarize yourself with the project management process._

Description

Replace the very old BSQ Explorer codebase by adding support for BSQ into Mempool.Space, and replace our current BSQ Explorers with Mempool Nodes so Bisq gains an integrated BTC and BSQ Explorer with Mempool Fee Estimation API support.

Rationale

Bisq's current BSQ Explorer sucks. Mempool Explorer is awesome. Having a modern explorer website for $BSQ will add a huge value to the Bisq infrastructure.
Bisq's current BSQ Explorer cannot display unconfirmed transactions. When users click on an unconfirmed BSQ transaction from within the Bisq app, the BSQ Explorer crashes with a strange NaN error. The Mempool explorer will simply show this as an unconfirmed transaction, not showing any BSQ details until confirmed, and display realtime TX tracking until it does get confirmed, letting users know exactly what is going on with their BSQ transaction.
Bisq's current BSQ Explorer backend often gets stuck due to various issues and is a headache for BSQ Explorer operators to maintain. Mempool Explorer is very stable and easy to maintain.
Bisq ops team will reduce monthly expenses by eliminating the BSQ Explorer Operator roles, since the Mempool Node Operators will gain built-in BSQ Explorer functionality.
Bisq will gain its own federation of self-hosted Bitcoin Explorers, so the Bisq app can be modified to use the same explorer for both BTC and BSQ transactions. This will reduce reliance on external TTPs such as Blockstream etc.
Bisq will increase its number of Mempool Fee Estimation backends, increasing the reliability of the Pricenodes to deliver accurate Fee estimation to all Bisq lite nodes.

Criteria for delivery

When the BSQ Explorer Operators reach Rough Consensus that Mempool has implemented all the tasks below and can replace the BSQ Explorers entirely, we can consider the project to be delivered.

Measures of success

When all BSQ Explorer Operators migrate to using Mempool, the Bisq users are happy with the change, and the role of the BSQ Explorer Operator is eliminated saving us money each month, we can consider the project to be a success.

Risks

Not much risk, since Mempool explorer is quite stable, and we can always go back to the current BSQ Explorer code if a major issue arises that can't be immediately fixed for some reason. The only risk I can see is that some transactions are incorrectly displayed until all the bugs are worked out, which can be mitigated with lots of testing before migrating.

Tasks

[x] Implement full BSQ support into the Mempool backend, including parsing the JSON data dumped from a Bisq node every time it changes on disk
[x] Implement full BSQ support into the Mempool frontend, including displaying the BSQ transaction metadata, for all BSQ transaction types, as it updates in realtime
[x] Must support all functionality of existing BSQ Explorer, and replace seamlessly including identical URL schemas, API endpoints, etc.
[x] Scope of BSQ support in Mempool can be expanded in the future to add new features such as displaying DAO vote results, DAO financials, etc. - @bisq-network/growth team might like this feature for integrating with Bisq main website.

Estimates

Dev: $3000 Ops: $1000

Notes

Mempool: https://mempool.space/ Mempool on GitHub: https://github.com/mempool/mempool

Emzy commented 4 years ago

Concept ACK

To replace the Bitcoin Explorer part it needs a Bitcoind + https://github.com/Blockstream/electrs running in the back-end. This needs a beefy server with > 1TB SSD. Would that also be needed for the BSQ Explorer part?

wiz commented 4 years ago

Yes, but by combining Bitcoin Node ($50) + BSQ Explorer ($100) + Mempool Node ($50) you get a total budget of $200 per month to run this beefy server which does everything all in one. It makes sense.

m52go commented 4 years ago

Scope of BSQ support in Mempool can be expanded in the future to add new features such as displaying DAO vote results, DAO financials, etc.

I assume this isn't a near-term / high-priority thing, but it sounds great.

softsimon commented 4 years ago

@wiz sent me a Bisq dao JSON dump and I have made some quick mockups to give an idea of how I would implement this.

Basically Bisq will be another optional module visible in the in mempool selection just like Liquid or Testnet currently. So heading to mempool.space/bisq will default to the familiar Latest bisq-transactions. Viewing individual transactions will have the same features and functionality with live tracking and fee information as on mainnet but with the additional Bisq DAO information.

wiz commented 4 years ago

Thanks Simon, I’ve assigned you to this project. Just to give you an idea how bad the current BSQ Explorers are, right now 2 out of 6 of them are stuck again today, despite our repeated attempts to fix them over the past few months. I’m really looking forward to replace that old code!

🔍 BSQ Explorers most recent TX block height as of Tue Jun 23 19:55:01 UTC 2020

bsq.ninja -> 636039
bsq.emzy.de -> 636039
bsq.vante.me -> 635989
bsq.bisq.services -> 636039
bsq.bisq.cc -> 635832
bsq.sqrrm.net -> 636039

wiz commented 4 years ago

https://mempool.space/bisq has an early WIP available for beta testing

wiz commented 4 years ago

The new explorer seems stable, so I have begun redirecting https://bsq.ninja/ to https://mempool.space/bisq as follows:

set $redirect_uri https://mempool.space/bisq;
if ($uri = /tx.html) {
        set $redirect_uri https://mempool.space/bisq/tx/$arg_tx;
}
if ($uri = /txo.html) {
        set $redirect_uri https://mempool.space/bisq/tx/$arg_txo;
}
if ($uri = /Address.html) {
        set $redirect_uri https://mempool.space/bisq/address/$arg_addr;
}
return 301 $redirect_uri;

wiz commented 4 years ago

Everyone seems pretty happy so I just merged the PR: https://github.com/mempool/mempool/pull/92

bisq-network / projects