Recent fee rises and TxQ issues

[donovanhide]

During today's large increase in transaction fees, possibly due to a certain tweet, rippled is spewing lots of the following logs. Market makers connected to this instance are receiving "Can not queue at this time." responses. Is this the intended behaviour? The market makers are trying to use low transaction fees to get their offers onto the books and to make money by keeping the fees low. The end effect of the transaction fee increment process seems to be that the market makers are getting priced out of the market, which is ironic :-)

2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction C92AC1CBCF48EDB1FDBB790CCFBD6EA74F5BDE6ACE2D27352FB74B5EF472AD98 fee is lower than end item's account average fee
2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction 2C5AF5FB72BA96A8CBBA6980D98ED42C6105627856F96DC5269726F46DAB1586 fee is lower than end item's account average fee
2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction 1701E6EAC50CA0D9994AED3BECF49E14FA6DCDC000F175A202820A7376B80A3D would kick a transaction from the same account (rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBX) out of the queue.
2017-Aug-28 22:47:22 TxQ:WRN Queue is full, and transaction 9DCDD2AF543932BD43954C20BD3365396A83AEF2F94339027C3DE572C6B8173E fee is lower than end item's account average fee
2017-Aug-28 22:47:22 TxQ:WRN Queue is full, and transaction F1F70AF66AEE4C497BB97522C6034BCEC3B3E8D92277D97D10F8BCB621142FC5 fee is lower than end item's account average fee
2017-Aug-28 22:47:23 TxQ:WRN Queue is full, and transaction 9696B9F9E3195423214B310B7F4885F0B90F032C4E57DCA4090BEC4F818682B7 fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Removing last item of account rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBXfrom queue with average fee of27297 in favor of 10982E55CC5219B0B4B90B5B2D205965196171F8C8A720800EF14D6B9F789C21 with fee of 446796
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction A1DEDCCF06002577E40DA3C2302674C68957B0F479264B66056D4D1457CC2EFE fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 7AB84471D805ADE13BE32F5E83506D14A35AF08FAD790AD8F4AAF244904A00C1 fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction E7EC553FB99FB7948005A09AA41AE94418D80109926E557B0A089571D39FEDBF fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 688C1E363C3B9E8391BEA044A9D57E8B45CAD6CC90DB2992C3B205588D62E03E fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 4361FE30EFBF320166B9289B8B4263D86D0381BD826D78A096DDE1E512F99E78 fee is lower than end item's account average fee
2017-Aug-28 22:47:26 TxQ:WRN Queue is full, and transaction C801A86FEA2707A656CA741DE727BD5D87F054F0F2830A5BE832245BE9A75B70 fee is lower than end item's account average fee
2017-Aug-28 22:47:26 TxQ:WRN Queue is full, and transaction 5FD6ED2C1B61AEC719E0CE0485E712AE4771B448F8112E09E9F4B648200A79E7 fee is lower than end item's account average fee
2017-Aug-28 22:47:30 TxQ:WRN Queue is full, and transaction AAE9D9E47BE9DD6B7B2648D277F3F9266C80224CE250042F846B0A1BFF5843EB fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction 22C24DBD28C4FAA2F36E584330A719C919D8FB0DCA57407613663D9933DE4540 fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction AD735AF3B1298BDF5B355FE922BA318977D5FEA2C851A68E8DF9E610C9C377A1 fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction 40B59A6B2F425ACFAD39CA4F1458031993B39DBDEEBBAF137138A57643B727F7 fee is lower than end item's account average fee
2017-Aug-28 22:47:32 TxQ:WRN Queue is full, and transaction A749A097EBD0EC4B2F7F47F0D1393102A332DF4369D6336E2225DE8815F568B7 fee is lower than end item's account average fee
2017-Aug-28 22:47:32 TxQ:WRN Removing last item of account rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBXfrom queue with average fee of27469 in favor of 9C888D0ACA02C4D15AAE2253433EEF82B257C6B556AAC6BB99B9AA4E1DBF3F14 with fee of 38400
q2017-Aug-28 22:47:33 TxQ:WRN Queue is full, and transaction 6C0A59D1EB3E48494A30AEDFA6C85D3B6ABDABCA117B96DE67B344412551A353 fee is lower than end item's account average fee
2017-Aug-28 22:47:34 TxQ:WRN Queue is full, and transaction 50F231BEAC67BAB309FCBA3A19E2198A7183DC73339F34BD0D274AEF506E22F8 fee is lower than end item's account average fee
2017-Aug-28 22:47:34 TxQ:WRN Queue is full, and transaction 5DB0C39C7AA8F82D385D5A6B2709566989A18F2C83D86321D221BE8497F03F48 fee is lower than end item's account average fee
2017-Aug-28 22:47:35 TxQ:WRN Queue is full, and transaction 38B08A4E6CE70ABBE29053344069600C91A44BA9F8756AD4BB550874B603ABF6 fee is lower than end item's account average fee
2017-Aug-28 22:47:35 TxQ:WRN Queue is full, and transaction A32A8EF77A14D6E1039769DB11F86569D6857BE3662EC517F4168653C7BDD550 fee is lower than end item's account average fee
2017-Aug-28 22:47:36 TxQ:WRN Queue is full, and transaction C234E7971A60621DE677272426217020F656ACB14B305C2914E6DDD101C9A470 fee is lower than end item's account average fee
2017-Aug-28 22:47:36 TxQ:WRN Queue is full, and transaction 6BE17EAA6D8C93BBB1DCB4D39800B005BDCD44BF4C662AD0BBE9716852FD8B54 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction 936F4E461768015C8F49C3297DAF4DC53EC628B9D6EBEE8BAE62A598CD076731 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction B002C8025DA8A38029F45AB900B8017CABF655EE692323FFAB7063D15E684FC9 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction 89C6E7F4C12A80EF58E063D72E863B044A04D2B995F0DB8FD5543E296F494A5D fee is lower than end item's account average fee
2017-Aug-28 22:47:38 TxQ:WRN Queue is full, and transaction 77658D10FE4E7285922EC58880D1C6BDA1D7C1FB16C8D303F6DBF7641FCE50B9 fee is lower than end item's account average fee```

[ximinez]

@donovanhide Thanks for the feedback and your question. As far as we can tell, yes, everything is processing exactly as designed. To prevent runaway resource usage and/or abuse, the queue was implemented with a size limit of 20 ledgers' worth of transactions (https://github.com/ripple/rippled/blob/develop/src/ripple/app/misc/FeeEscalation.md#other-constants).

If the network was staying healthy, and there were no problems with consensus times (and I haven't seen any evidence of problems yet), then it's safe to presume that the expected ledger size was at least 50, which means there were at least 1,000 transactions in the queue, which is honestly pretty impressive!

[JoelKatz]

It looks like that's the case. For example, here's a fee distribution from one of those ledgers. The first column is the number of transactions, the second is the fee paid in drops:

  4 10
  5 12
 13 30
  4 68
 12 70
  1 100
  2 490
  1 800
  1 5191
  9 10500

You'll notice that several 10 drop transactions still managed to get into the ledger and the total number of transactions in the ledger was 52.

[donovanhide]

Thanks for the responses. I guess the real question is how do I choose a fee which can make a profit based on the information fed out from rippled. For example here are two recent ledgers and the fee amounts advertised during the time before each ledger closes:

Ledger   | TxnCount | Max Fee | Fees
32299043 | 56       | 9700    | 5550,5739,5932,6128,6328,6530,6736,6945,7157,7372,7590,7812,8037,8265,8496,8730,8968,9209,9453,9700,10
32299042 | 36       | 8265    | 5363,5739,5932,6128,6328,6736,7372,7590,7812,8037,8265,5180,5363

32299043 seems to end on a fee of 10 drops, while 32299042 ends on 5363. I don't really know how to interpret this data. A market maker needs to be able to adjust his/her prices based on external data and/or trades occurring on the rippled ledger in a timely manner. Without being able to alter his/her offers in a timely manner, money will be lost or opportunities missed. If the fees are too high then profits will also decrease. Finding the balance to this problem is difficult, especially when the fees are sustained at such high levels as they were yesterday.

[ximinez]

I don't recognize the format of that data, but if those are all the fees that were paid by transactions on that ledger, it means that any transactions that paid less than those amounts were either put into the queue or kept in the queue.

Remember, that when a new ledger is opened, transactions are first pulled off the queue in order from highest fee to lowest. As those txs are added, the open ledger fee can start rising. And if it rises above the fee paid by the remaining highest fee tx in the queue, that tx will be left in the queue. Then both the open ledger and the queue are opened up for submissions. Transactions can be put in front of other transactions in the queue if they're willing to pay more. It's a trade-off between fee paid and time waited, and I have no idea how to make that decision for you.

Oh, there is one additional set of data points I can offer. If you use rippled's ledger command with ledger_index: current and queue: true (IIRC), you can see all of the transactions, including their fees, in both the open ledgers and the queue. I don't know if that'll help you make a decision for your own fees, but that's what we can give you.

[donovanhide]

The Fees values list is a result of this calculation based on the incoming ServerStream messages recorded from the notification of a ledger close until the next ledger close:

https://github.com/rubblelabs/ripple/blob/d26222abf520402bbd257f92f7ecdc6f5371a15a/websockets/subscribe.go#L47-L49

Based on this information:

https://ripple.com/build/transaction-cost/#server-state

The basic problem statement for a typical market-maker might be: Every t seconds I want to submit n offers to replace existing offers from a set O which need to have their prices updated due to changed external data. These offers need to be updated within t seconds, otherwise the next batch will clash. I need to know what fee will accomplish this goal with a certainty greater than x percent. I can then use this value to adjust the offer prices accordingly.

I'd say that this is a usability problem. Currently, an understanding of a complex queue at the rippled end is required and the client needs to manage a further queue to dynamically feed the rippled queue. This is further complicated by the need to deal with cancellation of existing offers, which can get messy, as there is no way to mark an offer with a client-generated id, other than hacking the expires value.

If there was an estimateFee API call which could take parameters t,n and x and return a single fee value estimate, that would be very much more usable :-) Even better would be if this was a subscribe-able stream.

[donovanhide]

...
     {
            "LastLedgerSequence" : 32370764,
            "account" : "rG1dLiK6p4e6w3mvAVCEGAFv6TcbMN5wHY",
            "auth_change" : false,
            "fee" : "12",
            "fee_level" : "307",
            "max_spend_drops" : "12",
            "preflight_result" : "tesSUCCESS",
            "retries_remaining" : 10,
            "tx" : "F4011E477CA7EEE9ED3CBBD4D8A1768321578F4B6E6B76ED1DED6134A40DEA1A"
         },
...

Given the above section of the ledger current queue output, if that was my transaction, and I wanted to push it faster into a ledger, are you saying to re-sign it with a fee of 307 raised from the level of 12, and that would raise its chances closer to 100% of successful submission?

[ximinez]

https://github.com/ripple/rippled/blob/develop/src/ripple/app/misc/FeeEscalation.md#fee-level

[donovanhide]

Honestly, that link does not help or answer the question :-)

fee_level is 12/10*256

How do I use the information from the queue_data to help select a more appropriate fee?

[ximinez]

I think the crux of this problem you're describing is that you're asking rippled to predict the future actions of the other users / transactions / servers on the ripple network, and that's really just impossible without a well trained neural network or a time machine. All we can do is describe the past and present. If you want a (nearly) 100% chance of successful submission, the only real way is to pay the current open ledger fee (from the fee command, or using the load_factor from server_info). If you want to get into a given position in the queue, you can grab the ledger info as described above and pay a fee that will give you a fee level slightly higher than the transaction in that position, but there is absolutely no way I know of to predict or guarantee whether you'll stay in that position. The queue is essentially an auction, and you can't predict what others will bid.

[donovanhide]

I agree that a predictive statistical model of varying number of transactions combined with varying levels of fees yielding a chance of success output would be very useful, but it would certainly cost a large amount of XRP to produce the training data on the live network :-)

I'm more than happy to use the present values in rippled, which seems to be the open ledger fee. What I need to know is if the values in the server subscribe stream are appropriate. Strangely, I see the response is no longer documented here:

https://ripple.com/build/rippled-apis/#subscribe

except the brief excerpt:

Stream: server - Information about the server status, such as load_base (the current load level of the server), random (a randomly-generated value), and others, subject to change.

The formula used is here:

https://github.com/rubblelabs/ripple/blob/d26222abf520402bbd257f92f7ecdc6f5371a15a/websockets/subscribe.go#L35-L49

If I were to select the highest seen value after the last ledger close, and just before submitting my transactions, would that figure be a fair representation of the open ledger fee? Unfortunately there are four separate API responses where fee information is presented: server_state, server_info, server stream and fee and the docs are all a little vague about the fields in each.

[donovanhide]

Well, today it seems like: rpcyXFq6ArWozyYLUE52FnNDjWWcnGYTGe and rDfdaugfiyePPcNKHL2soyL3wDmya32Ya3 are having fun paying huge fees and inflating the transaction fee for everyone else, resulting in hardly any transactions going through and servers reporting they are full. I'm sure they will probably run out of XRP and it's non-malicious, but it is very much observable that the transaction volume topples when the fee rises to current levels, which does suggest most bots do not have a suitable fee selection strategy, which goes back to the original point of this issue.

[gituser]

I wonder if it's the same issue that rippled when I do send some ripples gives me transaction hash today but when I look it at https://ripple.com/build/ripple-info-tool/ it says tx not found.

What do you guys think? I do not see any errors in rippled log though.

[donovanhide]

Ledger 32683368 is a good example of dysfunction. 4 out of 5 transactions all paying over 2 cents worth of XRP to be in an empty ledger, preceded by not particularly full ledgers.

[gituser]

All day today I'm getting 'transaction not found' on ripple-info-tool whilst locally transactions are accepted.

I've resent now few of them, but I'm worried that I might send two times.

[donovanhide]

@gituser You need to check the validated transactions up until your specified LastLedgerSequence. The response you get when you submit is effectively useless.

[gituser]

@donovanhide how do I do this?

[donovanhide]

@gituser Attempt to interpret this:

https://ripple.com/build/reliable-transaction-submission/

into your own code :-)

[gituser]

@donovanhide the thing is the method I've been using for sending was working perfectly fine until today. Is there some network issues at the moment?

I can see also via account_info that my account is "validated" : false

[donovanhide]

@gituser I can't really help you debug your code, but this issue is that today, and on previous days, when some accounts raise the overall transaction fee, most other bots seem to fail to get their transactions into a ledger.

[gituser]

@donovanhide ok, I'm looking into that documentation you sent before to make proper submission of the transaction in Ripple network.

Thank you.

[gituser]

@donovanhide so I read about reliable transaction posting and it seems just too much hassle to implement all of that and I'm not sure that even after I do create the system which will check for LastLedgerSequence and re-submit transactions it will work better and won't give me more headache. In current system the transaction is only sent once. And if something goes wrong you've got to re-send it manually, but this also protects from accidental double sending.

So here's workaround I've used - I've just increased and made fee static to 0.5 XRP instead of using fee_mult_max parameter after this all transactions seems to be mined much more reliable.

[tuloski]

Yeah the fees are getting "ridiculously" high for market makers...they are not low enough to not be considered in the risk model. And it seems there are not too many transactions per second. Are the validators overloaded by some "malicious" attack that don't claim any XRP and the load_factor goes high? How is the load_factor calculated?

[donovanhide]

./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account | wc -l
366

Certainly lots of transactions in the queue, not many getting into ledgers...

[JoelKatz]

We've escalated and are investigating. We have definitely seen evidence of undesirable emergent behavior.

[donovanhide]

I'm seeing rpJrC46bPSUiWwTuuRGSiqHdSaCNWTFBZv consistently at the top of the queue with high fees. Oddly seems to lead to:

https://www.eobot.com/user/436753

Which suggests that maybe the ledger is being used for mining payouts and the payout bot has some bad fee calculation code?

Edit: seems that address is also linked to https://www.jubi.com/ ...

[tuloski]

What is the "latency" I see in rippled code to calculate if the server is overloaded? Where that time is coming from? I see there are 2ms removed for jitter so I suppose it is related to the communication with the other nodes, but it's weird that it is the only measure of load of the network. And then there are a lot of magic numbers (+8 *4) :)...hard to decode that code.

[tuloski]

The load factor I see is very high (minimum 3000 up to 400000 or more), so the fee is correctly calculated based on the loadFactor. Is there one of the validators having high load and then it propagates to the network? s-west.ripple.com where I connect has a loadFactorServer = 1, so it might be another one.

[donovanhide]

Queue even longer this morning:

 ./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account | wc -l
1019

Useful script:

 ./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account  | sort | uniq -c|sort -nr
    366             "account" : "rECfeqVQCWfUdFjV2E3teK3xNHtegLAX4a",
    102             "account" : "rBTp7LRcgYAXLDMUeKGCudamjienArx8eh",
     72             "account" : "rK3Grwc6uQS9G2Mfff1S5UUdnttCRVwtEN",
     62             "account" : "r9XRCuhi5uDZY9gPvpiCy6kWDjeH3wM4LU",
     49             "account" : "rpJrC46bPSUiWwTuuRGSiqHdSaCNWTFBZv",
     47             "account" : "r4aqu2zbAdX2QQ9bdeuGwifUMhtEeqx3pY",
     38             "account" : "rJapan2paeQA81SvYk1iEpgmvSh9Z7xaJs",
     37             "account" : "rabZ5dJdhqjJPnMzriaHp6VdLGMftPwXGh",
     35             "account" : "r9Gps6fB9YLuZ87rWx7M9TgLAGK2zsz5s6",
     19             "account" : "r9hNR8G9nARPL7LSAQoLv5EZdVjHBiXr7Q",
     12             "account" : "r9KG7Du7aFmABzMvDnwuvPaEoMu4Eurwok",
     11             "account" : "rsheL3o95iWwE8fnzVcNNeAS8iFwN6UwTw",
     10             "account" : "rwfGzgd4bUStS9gA5xUhCmg1J86TMtmGMo",
     10             "account" : "rnHXzwFLSen6hpTsKtVCNUpVKQsTZr8TeD",
     10             "account" : "rMQTGsAtFa1Aow2G9YRJabBwj1y1kTHLxN",
     10             "account" : "rEr3hxu5aim5tDWwH7H8BK47K91tR8c7FM",
     10             "account" : "rD4G6gtD2KwHqsRf7pcyA8r1neUzXT61ix",
     10             "account" : "r9Vv4yJeVFYqLBuZ85muKmxHvPzmZUDFMM",
      9             "account" : "rLAS7brcrD4H1tb2U7qFKEZAR5L2LuMxoy",
      9             "account" : "razorxL2wrHEheeJ7c3BZ14gAkciU7tMTd",
      8             "account" : "rhS2H7ETM3wBkFETvYycoUm9FEDYi44Pg4",
      5             "account" : "rP45RuF983ttNc6zT7WtthnTSDC1iaGhg1",
      5             "account" : "rH3uSRUJYoJhK4kL9x1mzUhDimKE2n3oT6",
      5             "account" : "rEBhG5cxEcmM9rTwZgsMyS7Hxs3xdLuJ6",
      4             "account" : "rnwmbU2Luj9ZWcisU6VGopMBsdMPhbq88B",
      4             "account" : "rnMCfd99pwRE8u4LxE43Z1pDzock2kXbLQ",
      3             "account" : "rUYMTVAPdEvTA46TZnwBwx6CGwoqg7HsG",
      3             "account" : "rspZhZwvM8y6iSJERn3LpfXoEHkAk5ond1",
      3             "account" : "rLJFygDvrqbGgwNew4WbiSTomMS95Tqqiq",
      3             "account" : "rLHtvB1kZimYJsDCqCX4iQxRtSAqqFtrA7",
      3             "account" : "rKd19spe7AruEeMFxwTG9jmXdBUPeHU9i7",
      3             "account" : "rHENTPe3nMDLGJd59e5a3tYH9BXz1nFhvA",
      3             "account" : "rcyS4CeCZVYvTiKcxj6Sx32ibKwcDHLds",
      3             "account" : "rbokx39budXTQJRz2oEQiNfe9FwHKbWRB",
      3             "account" : "rBeToNo4AwHaNbRX2n4BNCYKtpTyFLQwkj",
      3             "account" : "r5oibBjEvqt4TtLpbKx2qf7Bru5a8Y4DT",
      3             "account" : "r4WXyQXoWwk4zEKnWQwBNf563XyPXjhL7U",
      2             "account" : "rUSqPxBWpHGGYawAg2e2nnC7EEZAY5eSRw",
      2             "account" : "rUeFPRGNjtcbtezyQKKiDcS1eQyYLQ1gcr",
      2             "account" : "rsc1JP8UNUtpwxH2XN5xBiqTsP6Hu3Dmfi",
      2             "account" : "rPMzWvu7WZmWtTQb4bH9HdxGKHeWNy4vyN",
      2             "account" : "rpa9GNAHVoQQq2Z533ZsD8TX4Tsu5d9f4v",
      2             "account" : "rLELbUxAfArviMX1yWj4ccmxnaHP159Q8",
      2             "account" : "rKiXhWVFvihz9zr1VqPxCifyGoCbwUuTS1",
      2             "account" : "rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq",
      2             "account" : "rEp4SYVfwQgGbWCJV4tBAgGNpG2KeiaY1W",
      2             "account" : "raL7tMvfQUL9sHQMRdwWpEJFRo5bxtzeiE",
      2             "account" : "r3CmdWT7sgBw9FossNwp5g8xdo7iBJwFMM",
      2             "account" : "r1eb6z9HtARaHSSz2hygEDi8yubA1bsJu",
      1             "account" : "rUFYLgGfX8CECpDcqbbC6CWbDg5BzD1EJk",
      1             "account" : "rpP2JgiMyTF5jR5hLG3xHCPi1knBb1v9cM",
      1             "account" : "rnVSYhajS62ZzSQu9crNNeibJUmFGWKnRc",
      1             "account" : "rND7fYoHXX8zFPreKNYWgQnvGj9dLMhSe8",
      1             "account" : "rKaVa3JczUZQehKEbFKjYqdc29UJjG3gQJ",
      1             "account" : "rJ2V5E9eXhgzRFwcWaKpeuVJytfmkTXztZ",
      1             "account" : "rH1GonJfbvgAiMeWzGFB59Wo75Yd5pPz7",
      1             "account" : "rDAtpYpng9FQugwGbZvcrcYZRZJ6jpzMx7",
      1             "account" : "rDAN8tzydyNfnNf2bfUQY6iR96UbpvNsze",
      1             "account" : "rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL",
      1             "account" : "raq3FG6ftAVXw7F9ePX2zGKfGzADc8SEdc",
      1             "account" : "raG6ntBY6yDUwLiaVU29pbirMTv375o9MZ",

[donovanhide]

Looking at the TXQ trace logs, I'm seeing lines similar to:

2017-Sep-14 09:32:08 TxQ:TRC Transaction 5C0B9B414422157B71DB8C0D95295F3E9A07BD2C89F0C8A06ED796001D781CA2 from account rP45RuF983ttNc6zT7WtthnTSDC1iaGhg1 has fee level of 2560 needs at least 1991940 to get in the open ledger, which has 147 entries.

The fee_level of 1991940 seems to come from here:

https://github.com/ripple/rippled/blob/fc640504ba90ef4323f8e845e8c264b81e2054b5/src/ripple/app/misc/impl/TxQ.cpp#L156-L162

Not sure that the maths behind this calculation is properly justified.

[tuloski]

@donovanhide isn't the fee calculated as base_fee * load_factor? BTW it's pretty hard to follow rippled code on fees. I gave it a shot yesterday but without comments in many parts of the code is very hard to follow.

[donovanhide]

@tuloski Yeah, the code is complicated and possibly only makes sense to the people who wrote it :-)

I think the TXQ fee and/or fee_level is treated separately from the actual fee for the open ledger. What I have noticed is that when I restart rippled, the TXQ doesn't immediately fill up to the high levels it had before the restart, which means there is a lot of local state floating around, which probably means this is a nightmare to debug :-)

Ed is obviously working on it:

https://github.com/ximinez/rippled/commit/ff660dc2b9d132b7b1646c560ffa458c387db48e

[tuloski]

In my idea the fee should be only the one to protect the network, i.e. the open ledger one. What's the meaning of having a local fee?

[donovanhide]

I guess the idea is that the TXQ protects the network from floods of low fee transactions, especially when the fee is high. Of course you could just write a custom client that connects to the other peers and sends the transactions no matter what the fee is. The TXQ is probably also meant to help the user get his/her transactions into a later ledger, if the fee is high now. But that is not working at the moment... Another option is to have a global queue, shared by all other peers, but then you aren't charging a fee for the distribution of those transactions which fail, but do consume network bandwidth.

[gituser]

Now even with 0.5XRP fixed fee transactions ain't going into the network. I'm getting transaction not found via rpc-tool here https://ripple.com/build/ripple-info-tool/

Even after resubmitting yet again completely new transaction it's still not validated.

As for transactions in the queue there seems to be not much: ./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account | wc -l 41

Any good validators out there which are reliable? I'm using

[ips]
r.ripple.com 51235

in the config..

What is happening? For months the system worked just fine.

[jnr101]

I think there is a difference per rippled server. I am using a private rippled. An I am now dividing the api.getFee() with 100 to get those lower transaction fees. Most transactions are now queued, but none of them fails. Most fees that I am using now are smaller then 0.001 XRP. So, apparently, different rippled's show different behavior.

[gituser]

I do use my own rippled instance, but it doesn't help.

Do you resend these transactions manually?

Do you use LastLedgerSequence on the submit ?

Also I do not see my transactions in the queue in the current ledger via: ./rippled -q json ledger '{"ledger_index":"current","queue":true}' |grep 'txhash'

[jnr101]

• No, I do not resend the transactions in any way
• I do not set the LastLedgerSequence parameter (thus using the default of +3 current validated ledger)
• I check the transactions via other tooling, they certainly made it in the ledger(s)

[tuloski]

@jnr101 if the fees are different on different rippled, then it's even more worrisome :( Are you sure? Can you connect to your rippled and to a public from RL and compare the api.getFee()?

[gituser]

Anyone had the situtation when unconfirmed (validated=false) account balance Sequence number is more than validated balance Sequence number?

It seems that's the problem for my main account to send out new transactions.

I've already added LastLedgerSequence into all my new outgoing transactions which I'm getting from server_info and adding to it 4, but it doesn't help transactions are stuck and not being validated.

Any suggestions how to fix the difference of Sequence numbers for validated/unvalidated states of the account?

[donovanhide]

@gituser Yep, I've seen that.. pass "validated" in the "ledger_index" field for the account_info API call:

https://ripple.com/build/rippled-apis/#account-info

[gituser]

@donovanhide I did that. The question is: why my particular account isn't able to send transactions into the network without a problem, why they do hang if they are sent from this account?

If I try from another account (where Sequence number is the same for both "ledger": "validated" and without that param) the transaction is validated almost instantly..

Or maybe my account was banned put on some limit could be that's the case?

[donovanhide]

It's because your transactions are stuck in the queue... There is some complicated, increased fee trick to overwrite transactions in the queue, but I've never tried it.

[gituser]

@donovanhide any link to that trick/technique ?

also I've checked the queue via account_info call it gives me for ledger_index: "current":

"queue_data" : {
         "txn_count" : 0
      }

Is there any way to list these stuck transactions somehow ?

[donovanhide]

Step 4 here: https://github.com/ripple/rippled/blob/develop/src/ripple/app/misc/FeeEscalation.md#transaction-queue increase the fee by 25%. There's obviously no point doing this if the transaction has all the same values.

If you share your account id I can tell you if your transactions are sitting in my queue, plenty are :-)

[donovanhide]

Easier still, here's my current full queue:

./rippled -q json ledger '{"ledger_index":"current","queue":true}' | grep account  | sort | uniq -c|sort -n
      1             "account" : "r3mcYA8MUfZ9RSKhbNYzfMUP7Jsq9vNBha",
      1             "account" : "r9Gps6fB9YLuZ87rWx7M9TgLAGK2zsz5s6",
      1             "account" : "rB1za2ZVgDnNB7u8LbVN61k5nCByBUtXCA",
      1             "account" : "rfKTnx7UE2Ab596Gan6DFrxeKtijQxr2nZ",
      1             "account" : "rG7eucYLyK39JjptdTksiKKdGEPGSFPeW3",
      1             "account" : "rGFVGdAo2iwJPZNkRqQ7EC5UEcdiFsBYoN",
      1             "account" : "rHmgCr7g5i425UmfYQH7EnBQhocG3oT756",
      1             "account" : "rKPNA16qJ9Vpymu4xeRq98c4Nsk8jwEZar",
      1             "account" : "rMhPVE4SnkcbbpgZwxqZAiv5sRTYh3Lfrf",
      2             "account" : "rJBpryFMKP7dPRjaVJSogkVHrf8ydoQMcx",
      2             "account" : "rLHtvB1kZimYJsDCqCX4iQxRtSAqqFtrA7",
      2             "account" : "rUZwBRmxtK9PwoJqAsgJg5P5was3Bd7wjA",
      2             "account" : "rwvCPrwkL8dr1pGfrk74v8FXus7APEd8Lk",
      3             "account" : "rEp4SYVfwQgGbWCJV4tBAgGNpG2KeiaY1W",
      3             "account" : "rPandaauXvGmajQAng65b3sByFeMWxX8en",
      7             "account" : "rnMCfd99pwRE8u4LxE43Z1pDzock2kXbLQ",
     10             "account" : "rPvVZ7jE5QzkiUhHFxf6s4EanMRqzGC4bs",
     34             "account" : "rK3VruB2uFg3XALfX9UffHEQmjXEyGWgfc",
     38             "account" : "rJM7t2aG7aXid9j8LcnC6X3rf9GhX3ZKgN",
     42             "account" : "rGHDkp1JG9ccieSVyvXywU7MxcXBAYSKDn",
     44             "account" : "r9KG7Du7aFmABzMvDnwuvPaEoMu4Eurwok",
     46             "account" : "rL3iwiaRjQoPXZD1U7hNGSsSj3sj3CoYNg",
     93             "account" : "rusty54REyh8LnsupyXD61WEbnoAjcKwb",
    105             "account" : "rhTqdCG4HAM9uFeiQAtfpH9uKjK4VBxGeT",
    115             "account" : "rGskymq2eMkCopmzP9od3gfRJrC5V193x1",
    131             "account" : "rabbit1eyDGsiWzRH4wL4vnk7giSxWyMeb",
    200             "account" : "razorxL2wrHEheeJ7c3BZ14gAkciU7tMTd",
    312             "account" : "rJapan2paeQA81SvYk1iEpgmvSh9Z7xaJs",

[gituser]

There is no my account in your listings, furthermore I've checked on my rippled using the same query as you did and there is no indication of my account as well. It's super weird. I'm sorry but I don't want to affiliate myself with this particular account in public.

I can't even determine TXHASH-es which are stuck :/

[donovanhide]

Are you getting tefPAST_SEQ,terPRE_SEQ,telINSUF_FEE_P or terQUEUED responses when you submit? It's kind of irrelevant, because tx's with those submit responses can succeed anyway, but might give you a clue as to what is going wrong...

[gituser]

No I don't. I'm getting tesSUCCESS for the new transactions.

The old ones of course I can try to find in the history, but I'm not running full rippled node, I've set pruning to 2000 there so old data is being deleted and also by looking via https://ripple.com/build/ripple-info-tool/ on all those transactions I'm getting Transaction not found.

EDIT: I've just checked the transaction which I sent earlier (about 1.5 hours ago) and it's not found anywhere neither on the tool neither in my local rippled instance.

[donovanhide]

You can imagine the Ops department in a bank phoning the dev late at night saying the exact same thing :-) There are definite usability questions about the transaction queue... Batch submission with atomic guarantees would go a long way to simplify the user experience.

My best advice is just wait for the fix; my market makers cannot operate at a profit with the way things are currently working.

[JoelKatz]

I can see also via account_info that my account is "validated" : false

That just means that you are asking the server for the most recent information, which is unvalidated. Most rippled queries default to querying the open ledger. Typically this gets you the most current information, taking into account even unverified transactions. That's usually what you want. (For example, if you submit a transaction to send XRP to someone else, you don't want to see that you still have the XRP just because the transaction isn't fully validated yet.)

Unfortunately, at the moment, open ledgers are diverging a bit and querying the open ledger can show you things that are less likely to correlate with future network states. Querying validated information is always safe unless you're querying a misconfigured server or the network's consensus logic as a whole is broken.

My best advice is just wait for the fix; my market makers cannot operate at a profit with the way things are currently working.

Anything relying on being able to clear large numbers of transactions at low cost should probably wait for a fix.