Several transactions stuck in mempool for days despite sufficient fee (mainnet)

[deluca-mike]

Describe the issue

Transactions are reported as in the mempool for days (according to most explorer and our node), while some other public queries show them as mined. Yet there is no reason (from my understanding) for them not to be mined, as they are valid and have more than sufficient fee.

Can you reliably reproduce the issue?

(EDIT: I've since resolved this issue, so the below links may not work anymore)

See the most recent 6 (or more, depending when you're reading this) unconfirmed transactions here: https://blockchair.com/dogecoin/address/D6WpnsUFCVdKZp52PNGwT7Ei3NGkaGaoKp

Notice that all those transactions are perfectly valid, and are paying fees well in line with transactions currently being minded.

Take this transaction, for example: https://blockchair.com/dogecoin/transaction/41e90534bb9d59f48ed5a9887937e8a5de93344b600d87d49a00e73c91a35586

2 days unconfirmed, despite being marked as confirmed here: https://live.blockcypher.com/doge/tx/41e90534bb9d59f48ed5a9887937e8a5de93344b600d87d49a00e73c91a35586/

Now, obviously it's possible that blockcypher is incorrect, although I don't know how they could be, but it still doesn't explain why miners won't include those 6 transactions in blocks.

What version of Dogecoin Core are you using?

Dogecoin Core 1.14.2

Machine specs:

• OS: linux
• CPU: N/A (K8s cluster)
• RAM: N/A (K8s cluster)
• Disk size: N/A (K8s cluster)
• Disk Type (HD/SDD): SSD (K8s cluster)

Possible solutions:

We can try rebuilding and rebroadcasting each of them with a much higher fee, but that is somewhat of a last report here given the nature of our system.

[vertiond]

Your first tx '41e90534bb9d59f48ed5a9887937e8a5de93344b600d87d49a00e73c91a35586' did not meet the minfee of 1 DOGE. Each of your future txs depend on this first tx to be confirmed since you are using the same output as input.

[deluca-mike]

@Vertian Damn, I feel like an idiot. Didn't notice that. Thanks so much! (I realized the future txes obviously depend on that first one, but couldn't detect why that first one got stuck.)

Edit: Oh, and this means BlockCypher is broken. Not that that matters here.

[deluca-mike]

@Vertian, wait, looking back at past txes, then why did these get accepted?

https://blockchair.com/dogecoin/transaction/8ec874071ef7d1be6b66ca9c9ecc90149442751079c502c462d1caf8be779040

https://blockchair.com/dogecoin/transaction/81e93f42b4c37412509288529dcab0348d395d7b17d8f32724a31a4c0be46bac

To be honest, I never knew DOGE had a min absolute fee node policy. This seems to suggest some miners don't mind, and/or are inconsistently applying the rule as well. It also suggests some miner may eventually mine the unconfirmed ones, just as in the past. I won't wait to find out though, but still curious.

[vertiond]

Mainnet soft forked shortly after those txs and began to reject blocks with a base version of <= 3. Among the changes that are fully adopted now is an updated minRelayTxFee: https://github.com/dogecoin/dogecoin/commit/388fb79a3a78eae092c57e04c5a816697e14997d

[shibe2]

This minimum fee requirement should have been applied only to transactions with dust outputs. I believe, transactions with fee less than 1 DOGE but with all outputs more than 1 DOGE should still be confirmed eventually together with free transactions.

[langerhans]

Thanks for reporting this. We are aware of the issue here. As it has been pointed out the minRelayTxFee has been updated to actually reflect what it was intended to be. The problem is that some wallets will create transactions with fees that don't follow the rules. I know of at least Ledger that does this. Did you have Dogecoin Core create these transactions of yours that pay a too small fee? Cause in my testing this was never the case and the fees were calcualted correctly.

To summarize:

1 DOGE base fee
+ 1 DOGE per 1000 bytes
+ 1 DOGE for every output below 1 DOGE (to prevent dust)

[shibe2]

@langerhans Have you tested non-dust free transactions?

[deluca-mike]

@langerhans no, we used a bitcoin lib to build the transactions off-node, since our nodes aren't aware of our keys and addresses.

Are these:

1 DOGE base fee
+ 1 DOGE per 1000 bytes
+ 1 DOGE for every output below 1 DOGE (to prevent dust)

Definitive consensus rules for blocks with version >=4, or are they just node policies?

[shibe2]

@deluca-mike Node policies.

[deluca-mike]

@shibe2 So then are we saying nodes happily accepted and relayed our "too-low-fee" transactions, and now miners refuse to mine them, despite them being perfectly mineable (since these rules are not consensus-level rules)?

Also, when did these rules get into testnet?

I have some recent txes that also broke this "too-low-fee" rule that mined: https://sochain.com/tx/DOGETEST/7fe2075fe312180f4f93b8679853a40628ef0c252c7dd4b759422c09b3b0a0f4

[ara1313]

@shibe2 @langerhans We are in the situation where we have txs that are unconfirmed on the chain with the fee below 1.

It seems that we can't use bumpfee for the transactions that already went out and sitting unconfirmed on chain. How would you recommend to get those tx confirmed?

example: https://dogechain.info/tx/38be6b764da587333892bdbb0f20561f9ba2880f50d50ec3ce77bbae0660f8be

We also have large number of transactions stuck in the mempool with the fee below 1. What should we do with the txs stuck in the mempool?

After we upgrade to the 1.14.2, what happens to the transactions in the mempool? Is it going to reload?

[shibe2]

@deluca-mike In my opinion, low fee transactions should have been relayed and mined even according to new rules, albeit with low priority. For example, transaction 41e90534bb9d59f48ed5a9887937e8a5de93344b600d87d49a00e73c91a35586 doesn't have dust outputs, so dust rule should not be applied to it. One concern is that its input is from transaction made on the same day. AFAIK, such transactions have low priority among all free and low fee transactions. When such transactions are coming fast, yours may get stuck. But at least some of these transactions should be getting confirmations, and mempool should be clearing slowly.

[quentinlesceller]

Do we know at what block height the soft fork was activated? No shibe alerted BlockCypher of the incoming fork. Seems like some miners are still mining v3 blocks that we currently accept. We will still follow the longest chain though. We are working on a fix right ow.

[patricklodder]

Do we know at what block height the soft fork was activated? No shibe alerted BlockCypher of the incoming fork.

3465058

[shibe2]

TXID 30214018373d209dbe2629f26096bd3378e8e59af4d9627d58c486c992db1828

010000000601156a63b6377c17e532dc739a2e2830e10286476db5f2290fb524292304a819010000006a473044022046b0b4caeb96cd2f1188df791faaeb5ae32e4010c1f6cb5582c946455913099d0220477428a0d3027df9c33038c87203e3f7de89f068e0a030fe24195eada20d8405012102745c350e03e3ff697bcbc4e61890b2fb6884eb4cc6024457020ac6705bd594defeffffff02bff3be675e8266eb292d4592db68f915a2dbb65117bff3733291a4bb985428000000006a47304402200a7be6687e9eb3602554e3be6d9d10f889ab13fc4c5e91ac6dcc2000e6d646bb0220583a83c594b00342dd99e4d0983e38216b7d475b17dd84355d73b5ca017afb26012103fbdf935ff917e8ae41f4e8bb68e23443ad48762fa71e53b218da10a31ef54b3bfeffffff9a97ffafb98d68443e03606979c1d9546a036c33e7a93f621a9b2efdd7f20e62010000006b483045022100a8c6eee898365e1a9fc2e48468cd86950d4573dedf199e4f1452f52919a6406702201bfd08af75b8fdbd8eb635ff60f664dd902157e9712b0da112eb3f65ba62a6e90121030996b8b8a6784010c1862842c1c5b89432a1b91608b67928dd8c878f5675123efeffffff499fea6910b2ab30501a5526b85b24104066ec240879c931c5d6d6fe3a9184cc000000006b4830450221009ec3ce54808f15aee3041213d4328eefbd0ddcde00633f0603ec9ac7008fc822022009d28bf0b1786a5377b9f187f8b03d502a288e8022fa2e8a91641a07acc850020121027d385e84bdff8e1a4407b47165505b9a3afc859c8f2f4cd9d848ef2fdcb85931feffffff7650bb1effec491e7eed0895a1b09cf415713fd60a7c4508e29f48c3b37f6915030000006b483045022100f63cf3160a4547d195c0b954ad40d0e02f06952562d24a8f11ab15377c3a8cf002200272564b551c5a46cab3e554f83003bbad4f634296fb1ca5ab3907e1342b7cdc012103749d3626368c98197a7961779fcda88cae8b04887b007f94f8660c537918ff3efeffffffb7d629963147b927e9d8dc858dd54dbe6a6d32577c2fd4502dc82cdbaa2186e1040000006a47304402206767f5f81c0f7d1a5e5ffb57a49b65e5ce5b6dbb3fea01d7137e8460189392b602204ce2f81b2457a2ed189b6ed2d0cf644a4cc2a3ac872e49df895725ff84f9694c012103244c6c447d65bce3a82ed585665416eb856f416e6d98eb804ca8aa3e1c2d2dfdfeffffff062b146b72ce0500001976a9149ddee64d62b570573e22c212e4363227ebe6847588ac24536d8d120000001976a9147ce21adcb0338723ace58fc22aa4ea51818ed59188acc0ec57cb2f0000001976a914b8953fdb1ceb951bdd7611529356b53ed189abf688ac2017b45a4800000017a914d5bc9d23ad40359de4faa67e8571876bee7d6e2387005b8624180000001976a914932ff52e20bf56b23edec2fc48dff3e4e1e4fdc188acdeb68400000000001976a914b46f2713bb8137a7f75dfcdbe7d9eef367a53e3f88ac36df3400

It has all inputs confirmed. Size 1097 bytes. Fee 2 DOGE. It's not accepted by Dogecoin Core 1.14.2:

error code: -26
error message:
64: dust

Because it has sufficient fee, dust rules should not apply, I believe.

[langerhans]

That tx has not paid sufficient fee. It should be 3 DOGE cause there is one output below 1 DOGE, so the dust fee has to be added. What wallet created this transaction?

[shibe2]

Oh, I didn't know about it. Do we need this special rule? I think, you tried to simplify fee rules.

It's on reddit, I will ask what software they used.

[langerhans]

I summarized the rules in this thread https://github.com/dogecoin/dogecoin/issues/1650#issuecomment-722229742

[shibe2]

I see. But wouldn't it be better to remove this rule so to make calculation simpler?

[langerhans]

For whom? The user? They don't need to worry if their wallet software does the math. Or for the wallet? I don't see the complexity in checking the size and iterating over the outputs. The only thing where it gets a tiny bit harder is when the change output is below 1 DOGE. At that point it's cheaper to just roll the change into the fee cause you'd pay more in fees to get the change back.

[shibe2]

It's easy to overlook a special case, like I did.

By the way, the transaction is from https://freedoge.co.in

[bosomt]

any idea why this transaction got stuck ?

22 DOGE size 21583

https://doge1.trezor.io/tx/7f06a211f511639f3db1ed5372ee921b2ce6a6ede2eed8428154d18c50207bdd

[patricklodder]

any idea why this transaction got stuck ? https://doge1.trezor.io/tx/7f06a211f511639f3db1ed5372ee921b2ce6a6ede2eed8428154d18c50207bdd

@bosomt This transaction's input #145 is unconfirmed (due to insufficient fee), so this gets rejected by all peers that do not have fa15a355...19c2

[richiela]

https://dogechain.info/tx/97504c52e8fce7aaba830b0ceb6d7616981dd2001ddf116c25983949e776ebbe Size | 1,493 bytes Fee | 1.99999997

https://dogechain.info/tx/90c3f0a2f8d3f69b361d846a78cdd9877bc0a42050011f625544abb7bf341763 Size | 2,179 bytes Fee | 2.80585938

These two seem to have the proper fees based on the above formula but yet wont confirm

1 DOGE base fee

• 1 DOGE per 1000 bytes
• 1 DOGE for every output below 1 DOGE (to prevent dust)

Anyone know why and what we have to adjust?

[langerhans]

They would need 2 and 3 DOGE fee respectively.

[richiela]

Can you explain why? If there’s a min for of 1 DOGE + 1DOGE per an additional 1000 bytes... shouldn’t the fee for the first one be around 1.5 and the second one be around 2.2?

[langerhans]

That's not how it works. A 999 byte transaction would cost 1 DOGE fee. A 1001 byte transactions costs 2 DOGE, 2001 byte 3DOGE and so on. Plus the extra 1 DOGE per dust output.

[shibe2]

The formula is such that the recommended fee is always integral number of dogecoins. The transaction size component is rounded down. So 1493 bytes add 1 DOGE to the fee (1493 div 1000 = 1). Base fee 1 DOGE + size fee 1 DOGE = recommended fee 2 DOGE. For 2179 bytes the recommended fee is 1 + (2179 div 1000) = 1 + 2 = 3.

[richiela]

Hmm ok... this is def off script from most btc derived coins... we’ll make changes accordingly. Thank you for the clarification.

[shibe2]

To me it looks like original Bitcoin fee scheme from the days when blocks weren't full.

[ReverseControl]

@patricklodder @rnicoll @michilumin Close this one.