Transactions not being broadcast to nodes in a timely manner/Long Transaction Queue

[toxicwind]

Repost of https://github.com/dogecoin/dogecoin/issues/217

http://jsfiddle.net/pb5pz/16/ Click search and see how transactions are being sent to the chain.

Example from transactions:

    {
        "account" : "",
        "address" : "DG1nr9XACZjBpRVH51WvJMZmNVmzVxonZB",
        "category" : "send",
        "amount" : -23002.64265078,
        "fee" : -1.00000000,
        "confirmations" : 6,
        "blockhash" : "d45cbb87c2929a86124cc068bf7dd6d84d1ca10deb6d7190e94ae425ea122294",
        "blockindex" : 27,
        "blocktime" : 1391672465,
        "txid" : "44085683942bb775a06b023dd785a21d6cc1fe22abaf2e059a854f3789a1cd87",
        "time" : 1391655569,
        "timereceived" : 1391655569
    },

This transaction took over 4 hours and 40 minutes to _broadcast_ (the tx didnt even go out til then), even with a fee. "time" : 1391655569, vs "blocktime" : 1391672465, 16896 seconds == 4 hours 41 minutes 36 seconds

Example of this happening on live server, transaction log(updates every 3 minutes): http://dogepool.pw/pendingtransactions.txt There should be a block payout sent in a few minutes from when this is posted.

Some transactions take over 4 hours to send when you have a large amount of transactions, even when most of those transactions are send and receive transactions to change addresses.

Often the whole wallet balance will go into "unconfirmed" state(and wait for 3 confirms from the chain) even though the coin is being sent to addresses that belong to the same wallet. This seems to cause all future tx to not broadcast at all and just wait for the chain to get the 3 confirms, in which it again it rebroadcasts a random amount of tx(look at jsfiddle).

Things already tried: 5 change= addresses set

http://bitinfocharts.com/dogecoin/address/DTzjPeJEirk3i5eQt3jtTBUk4dSjosMvTq http://bitinfocharts.com/dogecoin/address/D9X6eX1L5KFWA2PJ4jAm68JoLNdmMSJaXw http://bitinfocharts.com/dogecoin/address/DEFVgt9FPxRPhzgHGzzrGZPZfnY9bDfybQ http://bitinfocharts.com/dogecoin/address/DDqgC76vsrG3yC14Vez94UY2hBxSLsQpyj http://bitinfocharts.com/dogecoin/address/DLJNapEx6hhKtKRmNnCWF1zRU7sgBcvTPs

1 change= address set fresh chain sync new wallet.dats recompiling with earlier boost version..

Possible causes: 3 Confirm wait for own wallet transactions CWallet::ResendWalletTransactions() Desynced chain [??? https://github.com/dogecoin/dogecoin/issues/229] RPC Calls from mining causing severe lag with sending to chain?

[toxicwind]

Ugh I hate to say this, but the live log didnt do that for the latest block as there wasn't as many transactions as normal as the coin generated was pretty low.. Got to wait for another block if you want to see how that happens live again. JSFiddle above still works of course. :(

Edit: scratch that, another block happened! Just wait another 50 confirms for it to pay out.

[azamatms]

Read through some bitcoin docs and it seems rpctheads is the culprit, not having enough of them that is. They mentioned that rpcthreads were being used to IO and wallet flushes while there weren't enough left for keepalive connections to peers and such.

Here is CPu load and usage when I set RPC threads to 32

http://i.imgur.com/Y96sCHU.png

here is the pool address http://dogechain.info/address/D5zkuK8ED66FEaVtMrXWq6Knt34FCdnJXZ

Once I set that I instantly had 8M doge in Auto Payouts come through. It still locks up but far less this time, I'll increase payout VM resources and up the threads to 64 with results I'll post later.

[toxicwind]

http://dogepool.pw/pendingtransactions.txt It's still doing it even with rpcthreads=128

[azamatms]

Yup, Still doing it here as well with 128 threads. Wallet flushes taking 500s to complete.

https://doge.rapidhash.net/debug.log

[toxicwind]

rog1121 pointed out to me that the ones going to change addresses have fee's of 0, is this causing them not to be broadcast for forever?

[bmbmikeg]

Same issue here. Payout transactions were getting stuck permanently. Downgraded to 1.4.1 and payouts are flowing once again. If there's any information I can provide to help resolve this issue, please let me know.

[toxicwind]

Been looking into it more and I very much think this has to do with change addresses, normally the way change should work is that you should be able to spend it no matter if it is confirmed or unconfimed, but for some reason it is making the whole change amount into "unconfirmed" as if it is actually sending the balance(thus waiting for 3 confirms), even though this is remaining in the same wallet account. I'm pretty certain this isn't the way this should be working.. Any ideas?

[langerhans]

I've been also looking into this now. I still don't have a solution though. What I wonder though is, that both of you @rog1121 and @toxicwind seem tosend nearly every transaction without a fee. I looked at many transactions now. On your textfile @toxicwind I did Ctrl+F for "fee" : 0 and it marked nearly all the transactions.
@rog1121: You misread the wallet flush times, it say ms, not s. Your mean time is around 500-600 ms per wallet flush. The mean time between ResendWalletTransactions() calls is around 26 minutes. That doesn't look too bad to me.
@toxicwind: I looked at the depug.log posted above and saw that change outputs get spent nearly immediately after being commited. Example:

http://dogechain.info/tx/8545ad470601b10d394de75a0f56ad196018f1e476db44cf5bd625a17aa5f23e

Interestingly, that tx still took 2h from commit to the wallet until block inclusion and was relayed 5 times until that.

Click the change output's redeemed at input and you'll see it has been spent even in the same block. Bitcoin just patched their client to specifically add an option which disallows spending of unconfirmed change. So what you say is the default behavior.

To sum it up, I don't think it's an issue with sending of transactions, but more with inclusion in blocks.

[toxicwind]

What is making those send without fees? Even though I don't think thats the problem... You can see later transactions with fees haven't sent. (some even have 3 dogecoin fees for some reason, lol.)

Check this out: http://dogepool.pw/pendingtransactions.php Click "search" and it will bring up all the transactions made. A 997k block happened about 7 hours ago and this is STILL sending transactions as of this post. A whole bunch haven't even been broadcast yet. Look at the Time Taken to Broadcast to Blockchain column, I feel like it definately has to do with the way things are being broadcast, if its fees id literally be okay with paying 1 doge for change transactions..

[langerhans]

The debug.log states otherwise. The transactions are broadcasted nearly immediately and are rebroadcast every ~26 minutes intil included in a block and verified by the wallet.
I don't know how the pool software sends transactions. That's not my field of expertise. But if it builds raw transactions, it must make sure to include a fee. Maybe the actual fee handling changed between 1.4 and 1.5... Then again, if your pool software is up to date it should be able handling this.

[toxicwind]

It's not the pool software not sending the transactions though? Is the lagged transactions causing orphans now as well? http://dogepool.pw/index.php?page=statistics&action=blocks http://dogepool.pw/pendingtransactions.php http://dogepool.pw/pendingtransactions.txt Guess I have to go back to 1.4.1 if this is the case.

[langerhans]

You're not getting my point I think. Orphans is another issue. What is you "paytxfee" config set to? Transactions with 0 fee will always take longer. I think 1.5 might be enforcing this stricter than 1.4 but I haven't checked this.
Also, if you think it's the chnage confirmation stuff, MPOS should be changed to use raw transactions instead of sendtoaddress. Then you can freely spend unconfirmed balances. But like I said, that's not the case from what I read.

[mohland]

Is MPOS still utilizing the local wallet file to handle everything through the Berkely DB?

Egads, if so... pool owners -- heh, start coding up another frontend. With a 1 min blocktime, you're going to have a bad time. This is not going to get better. TBH, this is a MPOS problem, not a dogecoind problem.

[toxicwind]

I don't think that's really fair, this is something that definately happens when large amounts of transactions are sent etc, how is it not a dogecoind bug, even if MPOS is creating it... I realize they need to implement sendmany, which they are doing: https://github.com/MPOS/php-mpos/issues/1238 That doesn't distract from the fact that this bug is happening at any point you make a large amount of transactions, this is probably happening to exchanges and any other service that utilizes a large amount of transactions...

Brainstorming: http://dogepool.pw/pendingtransactions.php Click search without entering any address and look at the time to broadcast(I'm saything this as "blocktime" the time when the blockchain saw the transaction, if the word "broadcast" is confusing this whole ticket) to the blockcjain If this is abot the 1 minute block time and the code has been made for a coin with a much higher block time, this obviously needs a change somewhere, it seems like it isn't a problem with other coins because they can broadcast a large amount of transactions before a block happens(10 minutes with bitcoin?), where this coin, it will broadcast only a few in the minute it is allowed to with this coin. Could some sort of transaction queue made? Obviously the way payments are being made with MPOS, it tries to pay out the users iteratively. Example of what I think is happening: if you make 100 transactions in one minute, it will send those transactions but then all future transactions after that will be put in a queue where only a few transactions are broadcast at a time and all future transactions have to wait over and over again to be resent.

I bet this is exactly what cryptsy's latest problem is about as well, thus the need for a new hot wallet. You guys are basically saying "who cares about the people running the services in my network, it works for the basic user!". Please think about the issues that have been happening and think of how much more they will happen if the popularity of this coin continues. We have already surpassed peercoin. Why ignore a obvious issue?

[langerhans]

We are certainly not ignoring that issue, there is a lot of discussion happening about this. It's just very very hard to find the issue. I think I will setup a testnet node and mine some coins, run a script that sends like 250 tx at once and see what happens. Way easier to find something when you can test it locally.

[toxicwind]

I noticed that my chain also seemed to be desyncing as the transaction queue got bigger, does the chain stop updating if the transaction queue gets too large as it floods it with transactions?

[toxicwind]

I conceded and moved back to 1.4.1, so the pendingtransactions.php above is now reflecting 1.4.1 again. http://dogepool.pw/pendingtransactions.php If you look, transactions almost send instantly (even with 0 fees I might add!) with 1.4.1... I have all the mining still going on 1.5, not that it matters much. This really needs to get fixed before some sort of hard fork has to happen or many of the pools/exchanges are going to have problems. I've already heard from multiple other pools that they are still on 1.4.1, sad, as the latest version just has more info given about transactions in general.

[langerhans]

I finally got around setting up a testnet node yesterday. I mined a few mil and then spammed my wallet with transactions. I don't know if it's any representative because I was the only miner on that net at that time. I generated a few thousand random addresses, made a script to send 250 transactions with a value ranging from 0 to 32786 at a time via the sendtoaddress rpc call.
The script runs took each around half a minute to complete, mind that I used the qt wallet to see them incoming. And the 1500 txs I sent went through to 3 confirms around 10 to 15 minutes later. Wallet flushes took around 120ms with a wallet size of 5.3MB. It sucessfully placed as much transactions as it could fit into each generated block.
Guess I gotta bloat the wallet a bit more to actually see performance decrease, so that's just my first observations.

I'll continue looking into it, and seeing that 1.4.1 handled all this better, I really wonder where the changes are...

[jetbalsa]

After a discussion in IRC we might of found the cause, needs testing, Most people compile without UPNP (Using USE_UPNP=-) Teamdoge compiled with, and I bet the devs are compiling with UPNP as well, TeamDoge's 1.5.1 wallet isn't having any issues with sending TX, most get posted within 10s

[bmbmikeg]

FYI:

I am experiencing this issue and I compiled with UPNP enabled.

[jetbalsa]

Odd, the something about our build is not the same, I also hear that netcodepool doesn't have this issue ether

[jetbalsa]

I dont know if it would do any good, but I can zip up our build dir and provide it for share

[bmbmikeg]

I'd be willing to do some testing with it.

[jetbalsa]

Ok, http://teamdoge.com/dogecoind.7z

Its the ENTIRE compile dir, should show everything

[azamatms]

@jrwr compiled with UPNP and adding 8 of my own private nodes as well. Still have the issue.

[toxicwind]

I hate to say I told you so, but, here's the hard fork I was talking about.

[ummjackson]

@toxicwind Was unavoidable, and your help is invaluable. Any leads on what is causing the transaction slow down? We're trying everything. If you want to hit me up on hello@ummjackson.com with suggestions, that'd be great.

[ghost]

https://github.com/dogecoin/dogecoin/pull/265

[leofidus]

This seems to be solved, closing