Report blocks that didn't include mempool transactions

[xaur]

The high level idea is to report blocks that didn't include transactions from the mempool.

If the transaction is in the mempool, and a new block doesn't include it, this is not great. If 2 blocks miss it, this is inconvenient. If the tx is hanging in the mempool for 3 or more blocks (while the blocks are not full), this is bad. Such delays degrade the UX of transacting, including use cases like exchange deposits/withdrawals and merchand purchases.

This is particularly bad for delayed ticket purchase tx because unlike regular tx, ticket tx are critical for the network. I have observed multiple streaks of 2-3 blocks that didn't mine ticket purchases after their split transaction was mined.

Detection of delayed regular tx requires monitoring the mempool and storing additional tx data not stored on the chain (like first seen timestamp). This information can be spoofed by the reporter (dcrdata in our case), but I still believe it is useful to have it.

Detection of delayed ticket purchase tx can be done based on the on-chain data, to some extent. This is because usually ticket purchase tx are preceded by "split" tx. If ticket purchase tx prepared by the split tx is not mined immediately in the next block, mark such block. Ideally, also show the amount of blocks that tickets are waiting by the time of that block. Since this is all on-chain data, this info cannot be faked by the block explorer.

[chappjc]

Good idea. We'll have to spec out exactly how it will work, with particular attention to the fact that mempool is a node-local concept.

[xaur]

How about starting from delayed ticket purchase tx, since they are most important (imo) and can be derived solely from the chain?

[buck54321]

I guess the figure we are actually measuring will be number of blocks between ticket split transaction and ticket mined (let's call this ticket delay). Tracking the data should be pretty easy. I'll think we'll face some challenges displaying the data in a way that makes sense, but that'll work itself out.

The question that I think we are trying to answer is Did this block's miner leave tickets in memool even though they included less that MaxFreshStakePerBlock tickets?. It's the translation from ticket delay to miner laziness that will present some challenges, especially for more recent blocks.

Did I get the motivation right? Either way, I think this is a great idea.

[xaur]

I guess the figure we are actually measuring will be number of blocks between ticket split transaction and ticket mined (let's call this ticket delay).

Correct. To clarify, the focus is not simply on measuring the delays, but on detecting such instances and getting an overall picture of how frequent they are. (I'd like to analyze it further and see if they come from the same faulty pool, but as explained in #1564 this cannot be determined with 100% guarantee).

I'll think we'll face some challenges displaying the data in a way that makes sense

I suggest to start with a separate page dedicated to "interesting events", similar to orphaned blocks or PoS invalidated blocks.

An initial idea for such page is to only show blocks "involved" in such ticket delays, which includes blocks having the ticket split tx, subsequent blocks not mining the tickets, and finally blocks that mined all tickets in that group.

Later this page can be extended to show all "delayed transactions", although I would suggest to retain the function to watch delayed tickets specifically, due to their importance.

The question that I think we are trying to answer is "Did this block's miner leave tickets in memool even though they included less that MaxFreshStakePerBlock tickets"?

Almost. The issue I observed is not that miners included some tickets while leaving some others in the mempool. Rather, the issue was that some blocks included zero ticket purchases, straight after the ticket split tx. Sometimes there are 2 or even 3 such blocks in a row, and I found it concerning. At first I thought that sometimes miners restart their daemons and lose the mempool, but this issue occurs too frequent in my view, and deserves investigation.

To illustrate, I can quickly find a few such blocks by hand:

• 385952 mined some splits, but the following 385953 mined zero tickets. 385954 mined those, so the delay is 1.
  • side note: interestingly, while tickets from all other split tx in 385952 were mined 2 blocks later, this weird tx looks like a split but the outputs were never spent. Hmm...
• 385758 mined some splits, but two next blocks mined zero tickets - those only came in 385761, yielding a delay of 2.
• 385736 mined some splits. Now click a few next blocks. Three subsequent blocks mined zero tickets, with only 385740 delivering those - a delay of 3.

If my understanding of the system is correct, if the blocks are not full and per-block ticket purchase limit is not hit, then most tickets are expected to be mined in the next block after the split.

[xaur]

Related chat discussion here.

[xaur]

Looks like all "bad" blocks mentioned above that didn't mine tickets sent their reward to DsiDegkW7HxidcUCNpRqpgskc9JFCkqwWeu, which to my knowledge belongs to UUpool.

[raedah]

Since this will require keeping historical / real time data on information (mempool) that is not available from the blockchain, it may be a good fit in dcrextdata which has multi node redundancy as well as data synchronization capability.

[xaur]

Cool, wherever this is implemented this will shed some light on how big is the issue.

[chappjc]

I have been seeing my regular transactions (neither spit nor ticket) sitting in mempool for multiple blocks with plenty of free space in those blocks. It's starting to seem like a big miner is not well configured.

In this case, the miner controlling Dsh3FPLz8LcgGomaLhbajSJYywDCQYU4cL7 skipped over transactions for two blocks.

[xaur]

Another instance: the mempool had about 10 tx and 80 tickets but blocks 419015, 419019 and 419021 mined nothing. They all have that DsiDegkW7HxidcUCNpRqpgskc9JFCkqwWeu as their block reward address.

Interestingly, 419020 also had that reward address but it mined a bunch of tx and full 20 tickets, unlike one block before and one block after it.

[xaur]

While investigating full blocks that turned out to be Treasury payments I noticed another instance of the empty blocks problem discussed in this issue.

On Feb 12, a series of 6 blocks that included Treasury payouts was delayed by 3 nearly empty blocks. The first two of them (422857, 422858) included just 1 coinbase tx and no ticket purchases, thus being empty. The 3rd one (422861) included just 1 tx (sending 783 DCR) besides the coinbase. All three again paid PoW rewards to DsiDegkW7HxidcUCNpRqpgskc9JFCkqwWeu. The other 6 normal blocks full of transactions used different reward addresses.

On Mar 16, however, there were 9 nearly full blocks with Treasury payments with no empty blocks in between. 2 of the 9 blocks sent rewards to that ...Weu address but were all packed with tx as expected. I assume that this entity had no choice but to fix their setup after the consensus change on Mar 13.