Recent fee rises and TxQ issues

[donovanhide]

During today's large increase in transaction fees, possibly due to a certain tweet, rippled is spewing lots of the following logs. Market makers connected to this instance are receiving "Can not queue at this time." responses. Is this the intended behaviour? The market makers are trying to use low transaction fees to get their offers onto the books and to make money by keeping the fees low. The end effect of the transaction fee increment process seems to be that the market makers are getting priced out of the market, which is ironic :-)

2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction C92AC1CBCF48EDB1FDBB790CCFBD6EA74F5BDE6ACE2D27352FB74B5EF472AD98 fee is lower than end item's account average fee
2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction 2C5AF5FB72BA96A8CBBA6980D98ED42C6105627856F96DC5269726F46DAB1586 fee is lower than end item's account average fee
2017-Aug-28 22:47:21 TxQ:WRN Queue is full, and transaction 1701E6EAC50CA0D9994AED3BECF49E14FA6DCDC000F175A202820A7376B80A3D would kick a transaction from the same account (rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBX) out of the queue.
2017-Aug-28 22:47:22 TxQ:WRN Queue is full, and transaction 9DCDD2AF543932BD43954C20BD3365396A83AEF2F94339027C3DE572C6B8173E fee is lower than end item's account average fee
2017-Aug-28 22:47:22 TxQ:WRN Queue is full, and transaction F1F70AF66AEE4C497BB97522C6034BCEC3B3E8D92277D97D10F8BCB621142FC5 fee is lower than end item's account average fee
2017-Aug-28 22:47:23 TxQ:WRN Queue is full, and transaction 9696B9F9E3195423214B310B7F4885F0B90F032C4E57DCA4090BEC4F818682B7 fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Removing last item of account rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBXfrom queue with average fee of27297 in favor of 10982E55CC5219B0B4B90B5B2D205965196171F8C8A720800EF14D6B9F789C21 with fee of 446796
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction A1DEDCCF06002577E40DA3C2302674C68957B0F479264B66056D4D1457CC2EFE fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 7AB84471D805ADE13BE32F5E83506D14A35AF08FAD790AD8F4AAF244904A00C1 fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction E7EC553FB99FB7948005A09AA41AE94418D80109926E557B0A089571D39FEDBF fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 688C1E363C3B9E8391BEA044A9D57E8B45CAD6CC90DB2992C3B205588D62E03E fee is lower than end item's account average fee
2017-Aug-28 22:47:24 TxQ:WRN Queue is full, and transaction 4361FE30EFBF320166B9289B8B4263D86D0381BD826D78A096DDE1E512F99E78 fee is lower than end item's account average fee
2017-Aug-28 22:47:26 TxQ:WRN Queue is full, and transaction C801A86FEA2707A656CA741DE727BD5D87F054F0F2830A5BE832245BE9A75B70 fee is lower than end item's account average fee
2017-Aug-28 22:47:26 TxQ:WRN Queue is full, and transaction 5FD6ED2C1B61AEC719E0CE0485E712AE4771B448F8112E09E9F4B648200A79E7 fee is lower than end item's account average fee
2017-Aug-28 22:47:30 TxQ:WRN Queue is full, and transaction AAE9D9E47BE9DD6B7B2648D277F3F9266C80224CE250042F846B0A1BFF5843EB fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction 22C24DBD28C4FAA2F36E584330A719C919D8FB0DCA57407613663D9933DE4540 fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction AD735AF3B1298BDF5B355FE922BA318977D5FEA2C851A68E8DF9E610C9C377A1 fee is lower than end item's account average fee
2017-Aug-28 22:47:31 TxQ:WRN Queue is full, and transaction 40B59A6B2F425ACFAD39CA4F1458031993B39DBDEEBBAF137138A57643B727F7 fee is lower than end item's account average fee
2017-Aug-28 22:47:32 TxQ:WRN Queue is full, and transaction A749A097EBD0EC4B2F7F47F0D1393102A332DF4369D6336E2225DE8815F568B7 fee is lower than end item's account average fee
2017-Aug-28 22:47:32 TxQ:WRN Removing last item of account rfepxFxQKMSwn1otQRk35TgXbfLX6p8JBXfrom queue with average fee of27469 in favor of 9C888D0ACA02C4D15AAE2253433EEF82B257C6B556AAC6BB99B9AA4E1DBF3F14 with fee of 38400
q2017-Aug-28 22:47:33 TxQ:WRN Queue is full, and transaction 6C0A59D1EB3E48494A30AEDFA6C85D3B6ABDABCA117B96DE67B344412551A353 fee is lower than end item's account average fee
2017-Aug-28 22:47:34 TxQ:WRN Queue is full, and transaction 50F231BEAC67BAB309FCBA3A19E2198A7183DC73339F34BD0D274AEF506E22F8 fee is lower than end item's account average fee
2017-Aug-28 22:47:34 TxQ:WRN Queue is full, and transaction 5DB0C39C7AA8F82D385D5A6B2709566989A18F2C83D86321D221BE8497F03F48 fee is lower than end item's account average fee
2017-Aug-28 22:47:35 TxQ:WRN Queue is full, and transaction 38B08A4E6CE70ABBE29053344069600C91A44BA9F8756AD4BB550874B603ABF6 fee is lower than end item's account average fee
2017-Aug-28 22:47:35 TxQ:WRN Queue is full, and transaction A32A8EF77A14D6E1039769DB11F86569D6857BE3662EC517F4168653C7BDD550 fee is lower than end item's account average fee
2017-Aug-28 22:47:36 TxQ:WRN Queue is full, and transaction C234E7971A60621DE677272426217020F656ACB14B305C2914E6DDD101C9A470 fee is lower than end item's account average fee
2017-Aug-28 22:47:36 TxQ:WRN Queue is full, and transaction 6BE17EAA6D8C93BBB1DCB4D39800B005BDCD44BF4C662AD0BBE9716852FD8B54 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction 936F4E461768015C8F49C3297DAF4DC53EC628B9D6EBEE8BAE62A598CD076731 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction B002C8025DA8A38029F45AB900B8017CABF655EE692323FFAB7063D15E684FC9 fee is lower than end item's account average fee
2017-Aug-28 22:47:37 TxQ:WRN Queue is full, and transaction 89C6E7F4C12A80EF58E063D72E863B044A04D2B995F0DB8FD5543E296F494A5D fee is lower than end item's account average fee
2017-Aug-28 22:47:38 TxQ:WRN Queue is full, and transaction 77658D10FE4E7285922EC58880D1C6BDA1D7C1FB16C8D303F6DBF7641FCE50B9 fee is lower than end item's account average fee```

[gituser]

@JoelKatz,

I understand the difference between querying unvalidated vs validated open ledger.

The question is how to reliably submit new transaction?

1) I've added LastLedgerSequence which I get from server_state according to documentation and I'm adding to the resulting value 4. 2) If I omit Sequence in the transaction it defaults to the unvalidated open ledger (account_info without ledger:"validated") Sequence 3) If I specify Sequence in the transaction from the validated ledger (account_info with ledger: "validated") it gives me error tefPAST_SEQ

So what should be the logic regarding 2) & 3) if I submit let's say simultaneously (at the same time) multiple transactions will be there a problem?

Also, another question: according to the documentation https://ripple.com/build/rippleapi/#transaction-fees there is maxFee parameter but If I specify maxFee the signing fails.

And another question: How to replace some transaction with Sequence number? I do get on submit tefPAST_SEQ on my local rippled and transaction isn't going to the network.

@donovanhide

You can imagine the Ops department in a bank phoning the dev late at night saying the exact same thing :-) There are definite usability questions about the transaction queue... Batch submission with atomic guarantees would go a long way to simplify the user experience.

That's very true.

My best advice is just wait for the fix; my market makers cannot operate at a profit with the way things are currently working.

Seems it is the only way, but I'd like to understand more about how ripple works in terms of sending/confirming transactions and why some are confirmed faster than others.

[jnr101]

@tuloski, I did some checks and wrote a very simple program only doing the api.getFee() for each ledger. C1 = Client@home_local_VM C2 = Client@remote_VM1 S1 = Rippled@wss://s1.ripple.com S2 = Rippled@remote_VM2

I tried all combinations. It turns out that each rippled gives different fees back. (not sure if that is worriesome, I can imagine it will also look at local load). When connecting to the same rippled it will give the same fees back. It was at first a bit confusing, but even when connecting to s1.ripple.com it sometimes gives different fees. This is because there are several rippleds behind that address. The factor of difference can be significant. Below are 2 connections to s1.ripple.com at the same time:

[15/09/2017 21:15:03] Connected.. [15/09/2017 21:15:06] Fee (0.10004165625) [15/09/2017 21:15:09] Fee (0.10004165625) [15/09/2017 21:15:13] Fee (0.098415) [15/09/2017 21:15:17] Fee (0.098415) [15/09/2017 21:15:21] Fee (0.10086000000000002) [15/09/2017 21:15:24] Fee (0.097606640625) [15/09/2017 21:15:27] Fee (0.097606640625) [15/09/2017 21:15:30] Fee (0.097606640625) [15/09/2017 21:15:33] Fee (0.098415) [15/09/2017 21:15:38] Fee (0.099226640625)

[15/09/2017 21:14:54] Connected.. [15/09/2017 21:14:57] Fee (0.152006625) [15/09/2017 21:15:01] Fee (0.15) [15/09/2017 21:15:05] Fee (0.151001625) [15/09/2017 21:15:09] Fee (0.152006625) [15/09/2017 21:15:12] Fee (0.15) [15/09/2017 21:15:17] Fee (0.15) [15/09/2017 21:15:20] Fee (0.152006625) [15/09/2017 21:15:23] Fee (0.14900165625) [15/09/2017 21:15:26] Fee (0.14900165625) [15/09/2017 21:15:29] Fee (0.14900165625) [15/09/2017 21:15:33] Fee (0.15) [15/09/2017 21:15:38] Fee (0.151001625)

The client (C1 or C2) do not make the difference in fees.

[jnr101]

could it be that the systems are busy due to increasing use of some of the new functionality, which then would use more system resources then perhaps anticipated? in that case it would be just organic growth and the fee-escalation should be loosened somewhat. Or maybe we're dealing with non-optimal implementation of the new functionality..(But I am just wildly guessing here of course)

[tuloski]

@gituser I'm using the MaxFee without problem...just the transactions don't go through because my max fee is often not enough and it fails with telINSUF_FEE_P

@jnr101 I don't get why 2 different rippled should give different fees. At the end the fee that matters is the open ledger, because you could modify your rippled code to "ask" you 0 fees, but then it would fail in the consensus. In your case your rippled was giving a lower fee and then the transaction was validated? So now I can say that whatever I thought to know about the fees is not true :)

[gituser]

@tuloski

I'm using the MaxFee without problem...just the transactions don't go through because my max fee is often not enough and it fails with telINSUF_FEE_P

I'm getting this when I try to sign transaction with MaxFee field inside tx_json structure: Field "tx_json.MaxFee" is unknown.

[jnr101]

@tuloski All I can say is my transactions validate with apparently a lower fee setting than what @gituser is using. I don't know why. But I can say that my instance of rippled is configured 'tiny' and running on relatively fast system. And perhaps of influence, I do not allow inbound connections on my rippled

You can try the test with the fees with this code:

'use strict'

const {RippleAPI} = require('ripple-lib')
var moment = require('moment')

const feeCushion = 1.2

const api = new RippleAPI({
  feeCushion: feeCushion,
  server: 'wss://s1.ripple.com' // Public rippled server hosted by Ripple, Inc.
});

function timestamp() {
  return moment().format('DD/MM/YYYY HH:mm:ss')
}

api.on('error', (errorCode, errorMessage) => {
  console.log(errorCode + ': ' + errorMessage);
});

api.on('ledger', ledger => {
  return api.getFee().then(fee => {
      console.log("[" + timestamp() + "] Fee (" + fee + ")")
  }).catch(console.error);
});

api.on('error', (errorCode, errorMessage, data) => {
  console.log(errorCode + ': ' + errorMessage);
});

api.connect().then(() => {
  console.log("[" + timestamp() + "] Connected..");
}).catch(console.error);

install/run with

npm install moment ripple-lib
node getFee.js

start two instances and when it connects to different server you will find different fees

For comparison here are some fee results from my own rippled:

[15/09/2017 22:08:05] Connected.. [15/09/2017 22:08:07] Fee (0.06402665625000001) [15/09/2017 22:08:11] Fee (0.06272662500000001) [15/09/2017 22:08:15] Fee (0.063375) [15/09/2017 22:08:19] Fee (0.063375) [15/09/2017 22:08:23] Fee (0.06402665625000001) [15/09/2017 22:08:26] Fee (0.062081625) [15/09/2017 22:08:29] Fee (0.063375) [15/09/2017 22:08:33] Fee (0.5360574375) [15/09/2017 22:08:36] Fee (0.06402665625000001) [15/09/2017 22:08:39] Fee (0.062081625)

[gituser]

Regarding: Sequence number issue it seems to be a regression in the v0.70.1, I've switched back to the v0.70.0 and a miracle - now the Sequence for both validated/unvalidated account_info is the same and I also can send now transaction with previous Sequence!

EDIT: hm, no, after some further testing there is a difference again between Sequence numbers for validated / unvalidated account_info even on v0.70.0.

[wilsonianb]

@donovanhide do you have maximum_txn_per_account specified in your config to something greater than 366? I'm not seeing any accounts with more than the default 10 queued transactions. https://github.com/ripple/rippled/blob/develop/doc/rippled-example.cfg#L535-L538

[donovanhide]

@wilsonianb

Yep, had it whacked up since Ed advised me to do so about a year ago :-) I submit batches of market maker transactions greater than 10 about once a minute:

$ grep maximum_txn_per_account rippled.cfg
maximum_txn_per_account=1000

[donovanhide]

I'm now seeing less transactions per ledger than a Chinese Bitcoin Exchange circa 1st October 2017 :-)

[tuloski]

BTW I'm seeing many transactions with low fees (0.000011) getting into the ledger. I don't get how is that possible...they might have their local rippled with low load and low fee, but then why don't they fail in the consensus? How can I retrieve the open ledger fee? I see mainly these accounts with very low fee: rUp8CvtneUUL6qYaxfZwUHg2cwYuo5jugG, rEp4SYVfwQgGbWCJV4tBAgGNpG2KeiaY1W, rD8LigXE7165r3VWhSQ4FwzJy7PNrTMwUq

[donovanhide]

Hi Ed (@ximinez)!

thought I'd give the 0.70.2 hotfix an early quick spin. The queue submission responses are a lot more enlightening than the old blanket telINSUF_FEE_P, which is great :-)

So, my market making bots now can submit transactions to the queue, and now strangely, exactly 10 from each batch always seems to succeed and the rest get queued locally, but do not succeed before the next batch, roughly a minute later. Is this the maximum_txn_per_account setting being applied at the validator end? Maybe I'm being premature, and 0.70.2 need to be deployed elsewhere :-)

[ximinez]

Is this the maximum_txn_per_account setting being applied at the validator end?

Most likely.

[JoelKatz]

We don't expect the fixes in 0.70.2 to have much effect until they're more widely deployed. We are still testing and finalizing the build.

[donovanhide]

Queues have been drained and local load_factor seems to be back to normal. Guess the fix has been deployed to the validators. Any chance of a tagged release?

[ximinez]

Soon™️

[ximinez]

0.70.2 has been released to master and tagged. Enjoy!

I'm going to go ahead and close this issue. If you have any further problems, feel free to reopen or open a new issue.

[donovanhide]

Nice work. Trading has resumed :-)

If you do get a spare moment after this slog, would be good to get some thoughts on building a local model for predicting a transaction fee for a range of possible transaction numbers in the next ledger:

https://www.xrpchat.com/topic/9809-questions-on-fees/?do=findComment&comment=98593