[Chapter] UTXO consolidation

[jnewbery]

https://en.bitcoin.it/wiki/Techniques_to_reduce_transaction_fees#Consolidation

https://bitcoinops.org/en/xapo-utxo-consolidation/

[harding]

Claiming this topic. ETA March 31st Sept 30th. Two Weeks^TM Rough outline (comments welcome!):

• Intro
• Basic idea: you only need one UTXO per transaction---any more than that increases costs. However, because of the design of your business or because you often create change outputs, you may often end up with many UTXOs that are below the value you typically use in transactions. If you consolidate several of these into a UTXO with a value you do typically use (or larger) using a low-feerate transaction, it'll be cheaper than just stuffing them in one of your regular higher-feerate transactions.
• When does it make sense to consolidate?
  • When you're acquiring UTXOs in a particular value range faster than you're spending them or all lower-value UTXOs combined. (I need to think about how to explain this well, probably includes some visualization such as a table.)
  • Changeless transactions: when consolidation can actually be less efficient than doing nothing (but requires smart software!)
  • Two inputs and zero outputs (changeless) is still more efficient than consolidation unless you're doing super-cheap consolidations (this assumes segwit and I need to verify this)
  • If the range of values in the feemarket is small (e.g. when we're basically at the minimum limit), you can't get enough savings from consolidation to make it worth it (I need to do the math to quantify what fee difference is the minimal to make consolidation worth it)
• How to get low feerates (keep an eye on the overall situation and use patient sends; this should just be basics; real meat should go in the fee estimation chapter)
• Concerns/considerations (I'm indeterminate on exact label until I've thought about these more)
  • Until consolidation txes confirm, that value isn't available to be sent to users
  • May reduce privacy (huge consolidations versus lots of small merges---I'm not sure how that plays out in privacy)
  • If feerates drop, you could've consolidated previously at a higher feerate than the present cost to spend an input (e.g. you lose money)
• Implementation
  • sort your UTXOs by value
  • select lowest n UTXOS (~2,500 max)
  • send to single output
  • use a low fee (144 or 1008 block estimate), signal BIP125, use RBF fee bumping if necessary
  • optional: great opportunity to use -avoidpartialspend type behavior to reduce future privacy leaks
• Conclusion/summary

[moneyball]

Strong outline! I can't think of anything to add at this point.

On Thu, Feb 28, 2019 at 4:48 PM David A. Harding notifications@github.com wrote:

Claiming this topic. ETA March 15th. Rough outline (comments welcome!):

• Intro
• Basic idea: you only need one UTXO per transaction---any more than that increases costs. However, because of the design of your business or because you often create change outputs, you may often end up with many UTXOs that are below the value you typically use in transactions. If you consolidate several of these into a UTXO with a value you do typically use (or larger) using a low-feerate transaction, it'll be cheaper than just stuffing them in one of your regular higher-feerate transactions.
• When does it make sense to consolidate?
  • When you're acquiring UTXOs in a particular value range faster than you're spending them or all lower-value UTXOs combined. (I need to think about how to explain this well, probably includes some visualization such as a table.)
  • Changeless transactions: when consolidation can actually be less efficient than doing nothing (but requires smart software!)
  • Two inputs and zero outputs (changeless) is still more efficient than consolidation unless you're doing super-cheap consolidations (this assumes segwit and I need to verify this)
  • If the range of values in the feemarket is small (e.g. when we're basically at the minimum limit), you can't get enough savings from consolidation to make it worth it (I need to do the math to quantify what fee difference is the minimal to make consolidation worth it)
• How to get low feerates (keep an eye on the overall situation and use patient sends; this should just be basics; real meat should go in the fee estimation chapter)
• Concerns/considerations (I'm indeterminate on exact label until I've thought about these more)
  • Until consolidation txes confirm, that value isn't available to be sent to users
  • May reduce privacy (huge consolidations versus lots of small merges---I'm not sure how that plays out in privacy)
  • If feerates drop, you could've consolidated previously at a higher feerate than the present cost to spend an input (e.g. you lose money)
• Implementation
  • sort your UTXOs by value
  • select lowest n UTXOS (~2,500 max)
  • send to single output
  • use a low fee (144 or 1008 block estimate), signal BIP125, use RBF fee bumping if necessary
  • optional: great opportunity to use -avoidpartialspend type behavior to reduce future privacy leaks
• Conclusion/summary

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bitcoinops/scaling-book/issues/12#issuecomment-468498367, or mute the thread https://github.com/notifications/unsubscribe-auth/AAT6AyhEkZiv1IFo_fW1LIPKxvV770hrks5vSHjugaJpZM4aPkon .

[jnewbery]

Two inputs and zero outputs (changeless) is still more efficient

Do you mean one output? If not, I'm confused.

Until consolidation txes confirm, that value isn't available to be sent to users

You could use your own unconfirmed output in your spend, or perhaps you could RBF batch.

This interacts with 'predictive fee management' (see comment in https://github.com/bitcoinops/scaling-book/issues/13#issue-402432533). Not sure if you think it's worth commenting on that here.

[harding]

Do you mean one output?

Yes, d'oh.

[Until consolidation txes confirm] you could use your own unconfirmed output in your spend, or perhaps you could RBF batch.

Although this is obviously technically true, I don't think it's practically useful. If you can send payments to your customers at the same low feerate you use for consolidations, then consolidations actually increase your expenses by using more block chain space than necessary to accomplish the payment. The only situation I can see those techniques being useful would be an emergency liquidity crunch where you were willing to negate the benefits of consolidation in order to get access to "stuck" funds. But for that, a simple RBF fee bump of the consolidation tx is easy and the chapter is already going to recommend ensuring you can fee bump to consolidation in case it gets stuck during a fee rise, so there's no additional implementation complexity to just reuse that existing recommendation for an emergency crunch.

This interacts with 'predictive fee management' [...] Not sure if you think it's worth commenting on that here.

Thanks for mentioning; it does seem appropriate to mention it here, but I'll have to think about the proper division of ideas between the two chapters. Perhaps after this chapter is finished, I'll sit down and try to create a sort of dependency graph between all the proposed chapters so that can ensure each chapter references only ideas presented in an earlier chapter, making both writing and reading easier.

[jnewbery]

Although this is obviously technically true, I don't think it's practically useful...

This all makes perfect sense. Thanks!

Perhaps after this chapter is finished, I'll sit down and try to create a sort of dependency graph between all the proposed chapters

That sounds amazing!