Make use of maker contribution to tx fee

[domob1812]

As far as I can tell, sendpayment.py does not make use (nor even display) the maker's contribution to tx fees. I can imagine the following change (maybe optional): Say that I want to specify a (total) tx fee of some amount myself as taker, e. g., 10000 satoshi. I also want manual order selection. Then, the screen that displays the offers could explicitly tell me about the fee contribution of the makers towards my desired (total) tx fee. E. g., if someone offers a join for 5000 satoshi but pays 1000 satoshi as tx fee contribution, the screen could show me an "effective" cost of only 4000 satoshi for that order. (Because the total cost for me including the desired tx fee is 9000 + 5000 = 10000 + 4000 = 14000 satoshi.)

If I understand the current code correctly, the fee I can set is always simply added to those of the makers. I think that it would be a nice option to specify the desired total fee instead and show the makers' contributions in the way described above.

Does that make sense? I'm willing to work on an implementation if yes.

[chris-belcher]

I agree with the displaying of miner fee contribution in the manual choosing of orders. Also agree with displaying miner fee contribution in the "chosen orders to fill" message.

Neutral/disagree with the concept of effective cost being coinjoin fee minus maker's miner fee contribution. It is best seen that makers pay for their own inputs and outputs.

Miner's fee are a difficult problem, as the taker doesn't know how many UTXOs the maker will send until he's already chosen which order to fill.

sendpayment.py was always meant to be a very light script on top of the Taker and Wallet classes, the true user interface was always planned to be an actual bitcoin wallet. However I can't think of a better easier way right now to improve the miner fee situation so I am neutral/agree with the idea of a price floor option on the total miner fee. Code it so the tx fee contribution and tx fee floor options mutually exclusive. Widespread use of the tx fee floor option might make the miner fee contribution by makers go to zero.

[domob1812]

I understand that sendpayment.py is meant as a light wrapper. Let me start with the two first points you agree on, namely displaying the offered fee. This should help, at least, advanced users somewhat.

As it is right now, the offered fee is basically insignificant, right? If I set my yield generator to offer no fee while my competitor offers one, there won't really be any difference. Or do some of the other scripts (like the tumbler) use the fee offers as well?

[adlai]

The fee is significant, now that low-fee transactions regularly take multiple blocks to confirm; at the very least, this makes tumbler.py time estimates rather inaccurate.

Another idea for consideration is for makers to define a fee contribution per input, which is their real impact on transaction size.

[chris-belcher]

Taker could use the maker tx fee contribution to choose between them, so provide pressure to keep the fee up. Trouble is the taker doesn't know how many inputs the maker will send in advance. Perhaps what @adlai says about fee-per-input or even fee/kB could do well.

[domob1812]

https://github.com/chris-belcher/joinmarket/pull/136 implements the display of fee contributions when picking orders. This is the part everyone agreed upon.

[adlai]

136 is a good start, but in light of the ongoing "fee increase attack", I've come up with a completely different approach: what if makers didn't contribute directly to the miner fee, and instead advertised the minimum fee they're willing to sign? This would work as follows:

• Makers advertise, in each offer, the fee which they expect to earn, and their fee standard.
• Alternative offer types for this approach could specify a lower bound for net miner fee in satoshi, fee per kilobyte, and/or priority.
• A taker, having picked a few offers, builds a transaction using each maker's inputs, with the miner fee defaulting to precisely the maximum out of the chosen offers' fee standards.
• If --fee was specified, but is lower than the amount necessary for some of the chosen makers, the script offers the taker to either increase the fee appropriately, or choose makers with lower fee standards (An initial version could simply fail with a helpful error message).
• By the time the taker sends the unsigned transaction to each maker, it should pass each maker's standard; if it doesn't, they don't sign (perhaps with a helpful error message).

We could still have offers include both the current miner fee contribution, and the fee standard advertisement... but why add extra noise? It becomes a superfluous degree of freedom, as any combination of the three could be specified by the two. Proof is left as an exercise to the reader, although I'd rather you work on implementation ☺

[CohibAA]

It's like the classic "Who should tip the waiter?"... everyone SHOULD, because it means better service for everyone, but left to choose, people seem to prefer to put the responsibility of the "tip" or fee, in this case, on the other party. I like the idea of educating the takers (and makers) as much as possible regarding their self-interest in higher miner fees, which might sometimes mean extra noise in the interface.

I do like the idea of advertising a minimum fee, but there is no guarantee makers wouldn't just set it to zero in order to try to take whatever piece of the market they can, at least in the current climate. That said, I hope you code up this flow, or someone does. :)

FWIW, I've written a yield-gen bot that lets makers configure an amount to contribute per utxo, and commented in some guidance on why they might want to set it to various levels. I think the combination of ideas, and more focus on payment interfaces should be the next areas of focus.

[adlai]

The makers are waiters. Miners are the bartender. The customer should tip everybody, and servers should focus on service, rather than settling between each other... That's the protocol's job!

Makers without a minimum fee standard in the current climate (posterity snapshot) will have a bad time, but I there should be a hardcoded minimum fee, in case the taker selects a bunch of cheap makers without standards.

[chris-belcher]

Fundamentally the problem is that takers want their tx to confirm quicker than makers do. I initially thought makers would also want fast confirmations because they want their money to work as hard as possible, but as the volume of the market isn't very high the time pressure simply isn't there. If I understand it right, I don't think the minimum fee willing to sign would help, it's essentially giving up and allowing the maker's miner fee contribution to go to zero. I'm not sure that's right, makers should pay for their own bytes.

Any hardcoded minimum fee can be edited. We've already seen exotic new offers being announced in the orderbook so it's clear people are prepared to play with the yield generator algorithm.

In hindsight, maker fee contribution being an absolute number was a mistake, it should have been in fee/kB.

There will be a chance of protocol coming up soon (see the newprotocol branch) so we'll have a chance to get everyone on any new scheme we invent.

I think we'd need to change the taker algorithm for choosing offers. If a maker pays less than what their bytes cost to mine the taker should consider that as a cost that they have to pay, and include that in the cost calculations. An option for a tx fee floor (expressed as fee/kB) should also be added so takers at least are able to control how quickly their txs confirm.

Also we can make use of estimatefee from bitcoin core. Aside: it would be nice if some blockchain explorers had a web api for estimatefee.

[adlai]

Makers can pay for their own bytes by offering a lower fee (including negative). My conclusion is that the current system has too many free variables (flat fee contribution? per utxo? does the maker now need to also advertise how many utxos will be input?), and also leads to a tragedy of the commons because a single maker can give zero contribution without having any noticeable effect on transaction confirmation time.

[domob1812]

@adlai's thought is exactly along the lines of my original post. The taker is the one who decides, in the end, about the total fee of the tx. He/she is also the one that has most interest in specifying the fee. Thus one could add an option for the taker to decide on the total fee, and see makers' contributions towards that as "discounts" for their fees.

A definite weakness of this system is, as pointed out, that the taker does not know how many inputs the maker uses.

[chris-belcher]

Sure, makers can offset their lower miner fee contribution by charging less for their coinjoin. In that situation the taker's tx fee floor option will be used to make sure the tx gets enough miner fees.

Maybe the other makers need to calculate the tx fees and refuse to sign if they think it will take too long to confirm? Which might break the tragedy of the commons, the other participants can punish abuse of their common.

If we move to announcing fee/kB then that's just one free variable right? which includes how many maker utxos will appear.

[adlai]

A definite weakness of this system is, as pointed out, that the taker does not know how many inputs the maker uses.

@domob1812 but that's the beauty of makers advertising their minimum fee standard for the final tx:

• taker sends !orderbook
• makers reply
• taker picks makers, sends each one a !fill
• makers reply with inputs
• taker builds tx, calculates precise standard for each maker, calculates minimum viable fee (or sends !fill to other makers), adds own input(s)
• makers get tx etc etc

[domob1812]

That's true. But in this scheme, a maker that sends lots of "dust" inputs can force the taker in the final stage to pay a huge fee. It would be nice if that could be avoided already at the stage of order selection.

[chris-belcher]

Isn't that advantage of maker advertising a minimum fee is also there right now with makers advertising their tx fee contribution? and would be there with advertising their fee/kB ?

[adlai]

Any maker with a positive (and larger than txfee/minsize) cjfee is not contributing to miner fees... maker contributions are fiction!

An improvement (and simplification) would be for the taker to send all applicable makers a !fill, and pick (whether randomly or cheapest-ly) from the ones that reply with a reasonable set of inputs.

[chris-belcher]

What do you mean by the first part? I didn't understand. If they contribute to miner fees they contribute to miner fees. Coinjoin fees are orthogonal.

I hope the second part doesn't happen. It would be very spammish.

[adlai]

First part: what we're calling maker contributions are in actuality contributed by the taker, with the notable exception of this poor chap. This is what I mean by having too many free variables: maker fees are currently determined by two variables than have an unintuitive interaction, rather than: "I need to pay this maker this much, and my transaction needs to have this large a miner fee".

What's spammish about the second part? It's all private communication between the taker and makers. FWIW, all offers in the book are pure fiction until a maker has sent you inputs, and even then, you don't know the maker owns them until they're signed. Rather than spam, I'd call this a more transparent approach, as it enables takers to include information about the efficiency of a maker's inputs when they choose which ones to join with.

[chris-belcher]

what we're calling maker contributions are in actuality contributed by the taker

Why? When you do the accounting it comes out of the maker's utxos.

The second part may not scale well, although you said only message makers you're interested in so it might be alright.

[adlai]

what we're calling maker contributions are in actuality contributed by the taker

Why? When you do the accounting it comes out of the maker's utxos.

How did you get that? The sum of a maker's inputs is less than the sum of that maker's outputs. The sum of the taker's inputs is more than the sum of the taker's output(s). By definition, the taker has paid the makers, and the miner fee. The fact that some indirection happened during the negotiations is ultimately irrelevant. The only maker (that I'm aware of) who made a net contribution to miner fees is "b3gn3g".

The second part scales much better if we accept that a determined spider can trivially coax that information out of the makers, and have them publish it all in the open from the beginning... but I'll take my battles one by one ☺

[chris-belcher]

I understand what you mean now. I think our disagreement is about that negotiation, I don't agree that the negotiation is irrelevant. It's important, if the negotiation doesn't succeed then no coinjoin will happen. The negotiation is what gives meaning to lines like https://github.com/chris-belcher/joinmarket/blob/master/lib/maker.py#L141 You could say the maker is earning from the coinjoin fee and then immediately pays some of that to the miner fee.

This discussion actually helps clarify somewhat. To help improve the situation to make miners contribute more, the default negotiating position of the taker should be changed to take into account the maker's contribution. And the protocol should probably be changed slightly to make this a bit easier.

[adlai]

The precise form of the negotiation is irrelevant from the blockchain's perspective, as long as everybody signs in the end. I strongly believe that we have a lot to gain from changing its form to the one I've described.

I also believe that protocol changes should be backwards compatible by adding a new order type for new protocols, rather than changing the existing behavior. JoinMarket now has 11 contributors, ~36 makers, and an unknown number of takers (over 100 subscribers); smooth updates are far preferable to forks.

[chris-belcher]

I've read your idea again, I misunderstood it the first time. I thought it was equivalent to miners always contributing zero, but actually it isnt.

[adlai]

Far from it. The current system lets makers set a floor for the miner fee, but they have to demand a cjfee at least that large (before they even start thinking of their own pockets). The proposed system lets them set a floor for the miner fee independently of their own fee. I could've explained it better, although we've also hashed out a few kinks along the way. Thank you for reconsidering it.

[chris-belcher]

With the fee market currently pricing blockchain space at 200 satoshi-per-byte for one confirmation, it's worth drawing attention to this issue again. Incentives in joinmarket for miner fees are all screwed up, right now the taker essentially pays for everything.

[chris-belcher]

I've had some thoughts on the best way to organize the market to deal with miner fees. One day I hope joinmarket will be used by people to save on miner fees as well as increase privacy, especially if and when schnorr signature aggregation is added to bitcoin. This scheme should align all the incentives in sharing out the miner fees in an equitable way. I use the word "equitable" instead of "fair" because I don't mean everybody will pay equally towards the miner fees. Different people have different preferences about waiting times, fees, etc.

At first I thought we should replace the current fee contribution value with a fee-rate instead of an absolute number. But that would still require using fee estimation algorithms, and lately I've been thinking about issue #715 which makes the point that those algorithms are often centralized, could be manipulated and are necessarily worse because they rely on past information rather than present information. So to understand this proposal first read #715.

The blog post in #715 suggests another way of doing things, which is to ask the user for those S, B, M parameters which define a curve of fee rate vs block height, which I'm calling the fee-rate-curve. Then start low and use replace-by-fee to bump the transaction up until it's confirmed or until it gets too expensive.

JoinMarket just needs to combine these S,B,M parameters for the many different people engaging in the coinjoin.

I propose this scheme. Makers publish these parameters:

1. S,B,M values that define the so-called "contribution fee-rate-curve", that's what the maker pays for their own inputs and outputs.

2. Maximum byte count that the maker contributes to the tx (sum of inputs, outputs and signatures)

3. Minimum total fee-rate-curve, also in S,B,M values. The maker won't sign a transaction with a cheaper fee rate than this.

Makers can use 3) to control their confirmation time and make sure their transaction doesn't get stuck. This will be most useful for patientsendpayment.

Now, Takers are likely to be willing to pay a higher fee rate than makers because of time preference, if a maker is happy with 200 sat/byte and a taker wants to be in a transaction that pays 300 sat/byte then the taker must make up the difference of 100sat/byte for every byte the makers send.

Takers can use 3) to control how much of this extra fee they will pay. What will probably happen is takers will assume the worst, that makers will always send their maximum number of bytes and choose which offers to fill accordingly. This will give an incentive for makers to structure their offers so that takers have more information, for example makers will have one offer with maxsize equal to their highest value UTXO and maxbytes set to the size of one UTXO, another offer with maxsize equal to their two highest value UTXOs added and maxbytes equal to two bytes worth of UTXOs. And so on.

Once a taker has chosen it's makers, it creates coinjoin transactions with that match all the maker's contribution-fee-rate-curves, sends information to the makers and waits for signatures to come back. Then broadcasts them slowly until one confirms. NLockTime can be used to make it impossible for a later-expensive transaction to be mined early, and then the transactions can be trustlessly given to someone else (e.g. the makers) and the taker can go offline.