Fee estimator doesn't account for covenant submarkets

[rsolari]

The hsd fee estimator is bcoin code that has not been updated for hsd, so it doesn't consider fee submarkets at all.

By submarkets, I mean that there are four limits for how many transactions can go in a block, each of which could fill up separately:

• the main market for block space
• a submarket for 600 "updates" which are OPEN, UPDATE, TRANSFER and REVOKE covenants
• a submarket for 600 "renewals", which are FINALIZE, REGISTER, and RENEW covenants
• a sub-submarket for 300 opens, which are just OPENs. Note that these 300 opens are a subset of the 600 updates

Note that the other side of the market, the miners who make blocks with the hsd template code, behave rationally. They pick the highest-paying transactions for each block that respect the covenant limits.

I'm planning to write a new estimateFee() method that takes covenant type as an argument and estimates fees to account for submarkets. Does that sound like a useful thing to add to fee estimation?

[tynes]

Concept ack

Good explanation of the different markets. I agree with your logic.

It would be ideal if an additional class was added that implements the current fee estimator API but with an incremented version:

https://github.com/handshake-org/hsd/blob/a90a45bb9ea2c267e47bbaa173da1f181123f35f/lib/mempool/fees.js#L810

Then it would be possible to run both fee estimators and we can determine how the two perform relative to each other. Also good to show customers that new code saved them a specific amount of money time.

[rsolari]

Sounds good. I'll use VERSION.

we can determine how the two perform relative to each other

I expect the new estimator would help with TX liveness. I don't think it will save on fees. It will sometimes pay higher fees on covenant TXs to make sure that they get mined.

[tynes]

It will sometimes pay higher fees on covenant TXs to make sure that they get mined.

Good point. So TXs that include certain types of covenants will pay more because there is less possible block space for them. This will prevent transactions from being stuck in the mempool longer than expected. :+1:

[pinheadmz]

I think this will turn out to be a really big task. estimateFee() isn't just simple math, there is an entire fee estimator class that interacts with mempool and chain, tracking transactions from mempool-entrance to block-commit. You may find that the policy object needs to be quadrupled to track each type of tx separately.

I was thinking about a website that would track the different fee rates based on covenant and I imagined compiling an exotic node with four mempools, each with its own estimator, with a controller that sorts the tx by type.

Also worth noting to add to the insanity: a single tx could have multiple covenants!

[tynes]

I remember reading someplace that the best strategy for Bitcoin fee estimation was to use RBF and increase the fee over time. Watching https://mempool.space/tv to gain an intuition on how the fee market fluctuates over time, it seems like it spikes very high when an organization dumps a bunch of transactions into the mempool (Bitmex) and then randomly the entire mempool will be cleared when two blocks are found very quickly after one another. It is certainly possible to sneak in transactions that pay very low fees, if your time horizon is long enough.

Since Handshake doesn't have RBF and fee estimation is a very complex problem, I'd be curious to know if bitcoind has updated the way that they do fee estimation since this code was ported to bcoin. It would be interesting to know if there are other fee estimation approaches out there that do not rely on RBF.

[rsolari]

I think this will turn out to be a really big task. estimateFee() isn't just simple math, there is an entire fee estimator class that interacts with mempool and chain, tracking transactions from mempool-entrance to block-commit. You may find that the policy object needs to be quadrupled to track each type of tx separately.

Yes. We could run four PolicyEstimators inside a wrapper. We could send each of them different subsets of the mempool and confirmed blocks. That's probably easiest, but I'm also digging into ConfirmStats to see much I'd need to change to do the covenant booking-keeping in the stats object.

Since Handshake doesn't have RBF and fee estimation is a very complex problem, I'd be curious to know if bitcoind has updated the way that they do fee estimation since this code was ported to bcoin. It would be interesting to know if there are other fee estimation approaches out there that do not rely on RBF.

In 2015, bitcoind wrote most of the current fee estimator. In 2017, they tuned and refactored it.

bcoin/hnsd's fee estimator was mostly ported from C++ to Javascript in 2016, so I wouldn't be surprised if the parameters have diverged.

[tynes]

Thank you for doing that investigative work. Fee selection is a known weak point of bcoin that hsd has inherited. I encourage you to open a draft PR as soon as you feel comfortable so that we can openly give feedback and such.

[rsolari]

@tynes @pinheadmz Thanks for your help so far!

I put together a proof-of-concept of covenant fee estimator here. If you get a chance, I would love to hear whether you think I'm on the right track or doing crazy stuff.