search

XRPLF / rippled

Decentralized cryptocurrency blockchain daemon implementing the XRP Ledger protocol in C++
https://xrpl.org
ISC License
4.5k stars 1.46k forks source link

The transaction is not validated for a long time in Rippled.(Version: v1.7.3) #4010

Open MinwooJ opened 2 years ago

MinwooJ commented 2 years ago

When I request tx rpc call, the "validated" field in tx is false for 2 hours.

{
  "result": {
    "Account": "rGoLduPEt9YYcYPzZaMSqmpNbrCfkDtfPP",
    "Expiration": 691705495,
    "Fee": "20",
    "Flags": 0,
    "LastLedgerSequence": 68064456,
    "OfferSequence": 67321150,
    "Sequence": 67321152,
    "SigningPubKey": "0349931A358790D295A9C3EDD9F69AB78B834A93691B48246F507237C123AA90E5",
    "TakerGets": "70000000",
    "TakerPays": {
      "currency": "58474F4C44000000000000000000000000000000",
      "issuer": "r4XSSBVGENvgUrZFijX8xvsJtp9Mi2UPz9",
      "value": "2296.96989875118"
    },
    "TransactionType": "OfferCreate",
    "TxnSignature": "3045022100FCA885F874FF69F1139CF20CA7769442DA81686BA7903F2E7EB2BEF10A2247A802200E656D9E7DBD03C65DDC0152C7503BA8AE67D05D506C95019FDD1E068D855D03",
    "hash": "11C1DADC2B6B1B31E09D6867CA282B1A190680C95FA1797F3698AB35492CD41E",
    "status": "success",
    "validated": false
  }
}

This is the response of server_info at the time.

{
  "result": {
    "info": {
      "build_version": "1.7.3",
      "complete_ledgers": "67969325-68067419",
      "hostid": "LNELIBNODE1509",
      "io_latency_ms": 1,
      "jq_trans_overflow": "147681",
      "last_close": {
        "converge_time_s": 4.003,
        "proposers": 32
      },
      "load": {
        "job_types": [
          {
            "job_type": "untrustedValidation",
            "peak_time": 11,
            "per_second": 26
          },
          {
            "job_type": "ledgerRequest",
            "per_second": 2
          },
          {
            "job_type": "untrustedProposal",
            "per_second": 11
          },
          {
            "avg_time": 1,
            "job_type": "ledgerData",
            "peak_time": 36,
            "per_second": 2
          },
          {
            "in_progress": 1,
            "job_type": "clientCommand",
            "per_second": 2
          },
          {
            "job_type": "transaction",
            "peak_time": 7,
            "per_second": 157
          },
          {
            "job_type": "batch",
            "peak_time": 18,
            "per_second": 80
          },
          {
            "job_type": "advanceLedger",
            "peak_time": 28,
            "per_second": 5
          },
          {
            "job_type": "fetchTxnData",
            "peak_time": 1,
            "per_second": 30
          },
          {
            "job_type": "trustedValidation",
            "peak_time": 14,
            "per_second": 6
          },
          {
            "job_type": "trustedProposal",
            "per_second": 15
          },
          {
            "avg_time": 20,
            "job_type": "heartbeat",
            "peak_time": 54
          },
          {
            "job_type": "peerCommand",
            "peak_time": 13,
            "per_second": 1131
          },
          {
            "job_type": "processTransaction",
            "per_second": 157
          },
          {
            "job_type": "SyncReadNode",
            "per_second": 68
          },
          {
            "job_type": "AsyncReadNode",
            "per_second": 481
          },
          {
            "job_type": "WriteNode",
            "per_second": 270
          }
        ],
        "threads": 6
      },
      "load_factor": 579.87890625,
      "load_factor_fee_escalation": 579.87890625,
      "load_factor_fee_queue": 1.203125,
      "load_factor_server": 1,
      "peer_disconnects": "52",
      "peer_disconnects_resources": "0",
      "peers": 10,
      "pubkey_node": "n94MoLMpkvfmWATeNL71an82LJ555Q7yjCuLCJpUZpPSL5xLsyNp",
      "pubkey_validator": "none",
      "server_state": "full",
      "server_state_duration_us": "167686766",
      "state_accounting": {
        "connected": {
          "duration_us": "280152313",
          "transitions": 11
        },
        "disconnected": {
          "duration_us": "1169028",
          "transitions": 2
        },
        "full": {
          "duration_us": "1096047942509",
          "transitions": 65
        },
        "syncing": {
          "duration_us": "131076957",
          "transitions": 58
        },
        "tracking": {
          "duration_us": "3087305",
          "transitions": 65
        }
      },
      "time": "2021-Dec-01 23:44:10.677768 UTC",
      "uptime": 1096463,
      "validated_ledger": {
        "age": 2,
        "base_fee_xrp": 0.00001,
        "hash": "3A9DD6ED702BC717D9D5513BE560D619A23F173E286E866BB380589190814F73",
        "reserve_base_xrp": 10,
        "reserve_inc_xrp": 2,
        "seq": 68067419
      },
      "validation_quorum": 27,
      "validator_list": {
        "count": 1,
        "expiration": "2022-Jun-01 00:00:00.000000000 UTC",
        "status": "active"
      }
    },
    "status": "success"
  }
}

And after 2 hours, the response of tx gives "validated" true.

I have no idea why the node could not validate tx for 2 hours...

This is the server logs at the time.

2021-Dec-01 23:43:02.489775863 UTC TxQ:WRN Queue is full, and transaction F6A345502D85B6C342765556519DA2636C571118F7441BF64F26957483D426C0 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.514414456 UTC TxQ:WRN Removing last item of account rP6XWZzXnNSsQRncXYoTP9Ya6dn8prNfVL from queue with average fee of 307 in favor of 7720C6A8F449BFBE3D9D4E4F2D3212B8556EA7DDFF73388E1C7C9D2DD11B5BA6 with fee of 512
2021-Dec-01 23:43:02.552322703 UTC TxQ:WRN Queue is full, and transaction DDABAB3D7F430CBA286C3CE135159B4CD5B9E6ECC960BEDAC3CEC6361CB19C3A fee is lower than end item's account average fee
2021-Dec-01 23:43:02.613409789 UTC TxQ:WRN Queue is full, and transaction 035481EB94DCCCD446E64728F714208A398385D17ADBC8E90308F1194D2A3E74 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.613541594 UTC TxQ:WRN Queue is full, and transaction 34C8AD3A5B16ED23C6FA07EE205FEB0CFCDA538AFFFAB8EF86092814EFF17880 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.618199898 UTC TxQ:WRN Queue is full, and transaction FCB59821A211ECCD5510D062363F47C885D374E5C9CCA75C7FC561BEA02C6728 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.629318908 UTC TxQ:WRN Queue is full, and transaction B5FEDF2E164DE248CDDE6FC8FC66AAA3A5472F56675D4A45EA90D4344586850A fee is lower than end item's account average fee
2021-Dec-01 23:43:02.643733933 UTC TxQ:WRN Queue is full, and transaction 5F4E56B148DDCCBF39C1BD6A827E2069278BD833A944CF4F127D6CAB800D3A0A fee is lower than end item's account average fee
2021-Dec-01 23:43:02.659502218 UTC TxQ:WRN Queue is full, and transaction 75AF60AE320069A46D141A999F58CB459BA0E1B436518130AEA47484598FCF1F fee is lower than end item's account average fee
2021-Dec-01 23:43:02.661382126 UTC TxQ:WRN Queue is full, and transaction 00B010247F371017D4C1EEA9F51A179C7266BF23077DEDEE3B0502D37E6D4E9E fee is lower than end item's account average fee
2021-Dec-01 23:43:02.661453145 UTC TxQ:WRN Queue is full, and transaction 2A5656FE4AB8D5ABB85B4185668817F4C61BCDE1A3228B7F9B3FE9F08BEA6B85 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.676234615 UTC TxQ:WRN Queue is full, and transaction 8C2756E7C3AED082024E4BEA3EBE24C3BF4C6BE06F48C0067B9C4830ACE99A4B fee is lower than end item's account average fee
2021-Dec-01 23:43:02.676315810 UTC TxQ:WRN Queue is full, and transaction 9CD18FDB381695E161120F1ED91089F57B6850BF310F118E83A7BC7D3DA48870 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.676365216 UTC TxQ:WRN Queue is full, and transaction 2F4E922B918F9F9E5309E720EA514F1AA5B1F175B7974F4BE6F1DB72FD8738C9 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.682901084 UTC TxQ:WRN Queue is full, and transaction B9ED414F2080B6A4CAA341B5C69B9643325481C9762936F1A42E4438BB4005FF fee is lower than end item's account average fee
2021-Dec-01 23:43:02.706287122 UTC TxQ:WRN Queue is full, and transaction 07A92330208937DCDEDD0ED4FDEF07F894C5846E4B6393FD8178E6C2B3CCDBD3 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.706369919 UTC TxQ:WRN Queue is full, and transaction 219D1FB8E7B078EC80E1413BC7C2D024C0AF2B435243D058996C8439643F7547 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.706437045 UTC TxQ:WRN Queue is full, and transaction 2666960FB2449E4B61DA577917C30F47A71521CE15B6FC2558C314CBB0B548F0 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.706486030 UTC TxQ:WRN Queue is full, and transaction 28657653F77024D9B83EEC48590346848061F93BA26F565EEE914B2C99A72FE4 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.714824049 UTC TxQ:WRN Queue is full, and transaction 71F6DB96C9CDC9ABF0A1DC62610D8B1F739E1C4119340BC5F1445512721DFB52 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.714910151 UTC TxQ:WRN Queue is full, and transaction 4AA1E994892779437579EEB28B0F4078E6D9564EB7FCFAACFD933E5F48E31002 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.725926363 UTC TxQ:WRN Queue is full, and transaction 104BD9F8F1856A488828347C804DB9D20B4016F9EA50DCC8677BBDA4FCF0CEE8 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.735118479 UTC TxQ:WRN Queue is full, and transaction 5A21D416BA996DF8A2338CD5736E8E319F54313D1504AC10AB4E62B603F21977 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.735185193 UTC TxQ:WRN Queue is full, and transaction 5E090F22C584EAF23B0A7D334ACEC5137FD98030A43BB4FF2E4EF19D993429E8 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.735888020 UTC TxQ:WRN Queue is full, and transaction 8F3229903DB6D8AA74C2D32EA2BFBD2EC692541DE4BF1E5AD633D840AC466C62 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.735977704 UTC TxQ:WRN Queue is full, and transaction CE341178F0876FEEF4C2F27FF5173A93985E2D841CE19EE813830C0CA7F75670 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.736032801 UTC TxQ:WRN Queue is full, and transaction D64207C97CA7FC81024CEA89BE4B94DE5BB7CA3624A20134A69BA55D7A56CB56 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.736093520 UTC TxQ:WRN Queue is full, and transaction FE5C20E5A5C5AEBF382C955D97479A741576A6D3A7AAF5A7B546074DDCC8950A fee is lower than end item's account average fee
2021-Dec-01 23:43:02.736309521 UTC TxQ:WRN Queue is full, and transaction B95F76E2DCDB863B819902D4131E23D895C57B2BCE5A1965D04B9F2CE144B668 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.745982310 UTC TxQ:WRN Queue is full, and transaction D1583BCDC6F94BB68D30F03B38D6B4506C6410D5EB652C0B0A462F8CF3C441E2 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.746070103 UTC TxQ:WRN Queue is full, and transaction EAC2C62BF5ADF0B418DCF58CD4A1EAFB871E1B48CBE40A5D0B3B7FE6AA6E8483 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.757509038 UTC TxQ:WRN Queue is full, and transaction 0A2537EE0CDF35291C26CDC02E5BFBFE81A3A59C1710FCC87200CDD7649663C9 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.763576900 UTC TxQ:WRN Queue is full, and transaction B9804E24EC7BB11F43134A4078C7DE364F7D91F350824BF56E9FB7C68BB6EAA0 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.763624739 UTC TxQ:WRN Queue is full, and transaction 3C7818502E0C3A5A71F102E912D8D9263A95B815F6B173A2D1AFFDDF062E33A1 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.776031434 UTC TxQ:WRN Queue is full, and transaction 3DCBAE63B2B31DDA5451D0A6BB678BDEB224E0F1ACC8BABC77A1F820356B23B7 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.783396942 UTC TxQ:WRN Queue is full, and transaction D15135D890437D1F9255409471942BA691C00E09810C55669E27783624BDD6DE fee is lower than end item's account average fee
2021-Dec-01 23:43:02.785189844 UTC TxQ:WRN Queue is full, and transaction 8E99389F6A7F564625502940C2ED8A933EBAB512F57CB17545A12E1824EE0D35 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.791927445 UTC TxQ:WRN Queue is full, and transaction 5920F74A7DBE06428C457E0C9A182CED2B2E7252E01C665127F5412CA7948387 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.801910136 UTC TxQ:WRN Queue is full, and transaction ECCB40C9987F8973281816A7FFBC935A28F4D9B4B609EA64BBB6A0C4E0035659 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.803054046 UTC TxQ:WRN Queue is full, and transaction DB552BA6CCFA59E0E800A1D55B7D4248770A6B749B38FE41777DBE9083B19FB4 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.946856781 UTC TxQ:WRN Queue is full, and transaction 30B18E96452548447048F5D548301804546CB3F14118DF88BE155384F169D2E2 fee is lower than end item's account average fee
2021-Dec-01 23:43:02.982249489 UTC TxQ:WRN Queue is full, and transaction A61E5B817CCC85D80A09CC13BDD952AFFF4242FAAEB4133D03EDFAE1E7384C7F fee is lower than end item's account average fee
2021-Dec-01 23:43:03.048925168 UTC TxQ:WRN Queue is full, and transaction 491ABCE71D59E2F65048F81B1FF2ACFF09F29CD313F96A28BFFD2098BD789D06 fee is lower than end item's account average fee
2021-Dec-01 23:43:03.081504828 UTC TxQ:WRN Queue is full, and transaction 1823AEEFA876317932CB2BFC693734010C6AA8898CA882C240E2013439679D56 fee is lower than end item's account average fee
2021-Dec-01 23:43:03.257667205 UTC TxQ:WRN Queue is full, and transaction 3C92E1865B78D08974F46F329B79DDA6512531C59FE1DAB177F49658C672B5DB fee is lower than end item's account average fee
2021-Dec-01 23:43:03.340651102 UTC TxQ:WRN Queue is full, and transaction AD21B745FE28EE578F204B40E671CC6DCD8A2031D9FF98B31977B76CED618516 fee is lower than end item's account average fee
2021-Dec-01 23:43:03.376818297 UTC TxQ:WRN Removing last item of account rGyaQGZsPJqaEZGc8aV71gH9vhsBnMMym3 from queue with average fee of 307 in favor of 46C1E93F5181F4C6EC785F16E6A64DA35327F863F97F053FF9E67EEB16AFDC1E with fee of 3814
2021-Dec-01 23:43:03.484471057 UTC TxQ:WRN Removing last item of account rLBmQAfBjA7EgGGhLsWQG2BDrzby3CmYqU from queue with average fee of 307 in favor of F26C4F1A91B8DD4F134DAED482F5AB2B5CE3F67E994EF7EC6741D2817D6F710E with fee of 2560
2021-Dec-01 23:43:03.503483818 UTC TxQ:WRN Queue is full, and transaction 29988C70009FFE2A7DA3E957CBD64D9DE6940FC4DF0A6B5F7E5E02C0F68FD8A7 fee is lower than end item's account average fee
2021-Dec-01 23:43:03.538552263 UTC TxQ:WRN Removing last item of account rfAQxbym2coQLt7FDPDps7E7GpAV1bvmoE from queue with average fee of 307 in favor of 8E7CEC5B0B74866B48A0DEBF457B09981122D2BA50B3273685A720CDA830FF7E with fee of 6144
2021-Dec-01 23:43:03.637217886 UTC TxQ:WRN Removing last item of account rPSvcmNAvc6DAZYX2w5WM4bcnLKSxyt8VY from queue with average fee of 307 in favor of E4875DFE226F9981EDBE06E95D4D99D714E2D881F6A2DA8ACFB74126F1009FEE with fee of 6144
2021-Dec-01 23:43:03.656880714 UTC TxQ:WRN Queue is full, and transaction 59D8F27F7927D261D69E10D0FA4FFCDE34A91882D1D09C4B5621A20C052EF2F8 fee is lower than end item's account average fee
2021-Dec-01 23:43:03.723959270 UTC TxQ:WRN Queue is full, and transaction 6BEA7F2123942453B4A1CD356031DD2937DE7943255435741F7538D41BBDB218 fee is lower than end item's account average fee

This kind of issue happened 3 times today.

wojake commented 2 years ago

This is the reason why most validators' queue are full + the recent high transaction cost: https://status.xrpl-labs.com/incidents/p4dg7m41cpzt

MinwooJ commented 2 years ago

@wojake Then what can I do for my rippled node..?

ximinez commented 2 years ago

Suggestions.

  1. Use a higher fee. Transaction submission rates are extremely high right now, but the queue is doing it's job of prioritizing the highest paying transactions. The more your transaction pays, the more likely it will be validated sooner.
  2. Use a smaller gap on the LastLedgerSequence field. I don't know what the current ledger sequence was when you submitted the transaction, but a smaller gap gives you a definitive answer more quickly. Note that this does not make it more likely that your transaction will be validated - it merely lets you know sooner that it is not going to be validated.
  3. Ensure your server is staying up to date. The server_info you provided indicates that your server is up to date now (primarily because of the 'validated_ledger.age`), but if it falls behind, you may simply be querying old data when looking for your transaction.