Some objects can not be encoded, resulting in "not valid hex" error.

[nixer89]

I am using: "ripple-binary-codec": "^1.1.2",

I am currently trying to "encode" some ledger_data objects from rippled. There are a many objects which can not be encoded. The error is:

[2021-06-08 19:58:17] [LOG] Error: 283 is not a valid hex-string at Function.UInt64.from (..\node_modules\ripple-binary-codec\dist\types\uint-64.js:60:23) at ..\node_modules\ripple-binary-codec\dist\types\st-object.js:129:56 at Array.forEach () at Function.STObject.from (..\node_modules\ripple-binary-codec\dist\types\st-object.js:128:16) at serializeObject (..\node_modules\ripple-binary-codec\dist\binary.js:59:32) at Object.encode (..\node_modules\ripple-binary-codec\dist\index.js:24:12)

The above example occurs when encoding following ledger object:

{
  Balance: { currency: 'GCB', issuer: 'rrrrrrrrrrrrrrrrrrrrBZbvji', value: '0' },
  Flags: 2162688,
  HighLimit: {
    currency: 'GCB',
    issuer: 'rBfVgTnsdh8ckC19RM8aVGNuMZnpwrMP6n',
    value: '0'
  },
  HighNode: '283',
  LedgerEntryType: 'RippleState',
  LowLimit: {
    currency: 'GCB',
    issuer: 'rhRFGCy2RJTA8oxkjjtYTvofPVGqcgvXWj',
    value: '2000000'
  },
  LowNode: '0',
  PreviousTxnID: 'C0C37CE200B509E0A529880634F7841A9EF4CB65F03C12E6004CFAD9718D6694',
  PreviousTxnLgrSeq: 24695242,
  index: '0000041EFD027808D3F78C8352F97E324CB816318E00B977C74ECDDC7CD975B2'
}

There are many more objects which can actually not be encoded - mostly (a lot!) RippleState and DirectoryNode objects. In the above example, the "HighNode" value seems to be the culprit. (same for LowNode in other objects).

An example for an error encoding a DirectoryNode is:

{
  Flags: 0,
  IndexPrevious: '2',
  Indexes: [
    'A3454ACED87177146EABD5E4A256021D836D1E3617618B1EB362D10B0D1BAC6A',
    'E1ED9E8D280BBE0B6656748FD647231851C6C650794D5E6852DFA1E35E68630F'
  ],
  LedgerEntryType: 'DirectoryNode',
  Owner: 'rHzDaMNybxQppiE3uWyt2N265KvAKdiRdP',
  RootIndex: '0CB3C1AD2C371136AEA434246D971C5FCCD32CBF520667E131AB7B10D706E752',
  index: '0B4A2E68C111F7E42FAEEE405F7344560C8240840B151D9D04131EB79D080167'
}

[natenichols]

hi @nixer89, thanks for opening an issue!

You're absolutely correct, it looks like HighNode is the issue (in the above example). Looks like an issue w/ constructing UInt64 types from string hex.

What is the ledger_index (or ledger_hash) of your ledger_data request?

[nixer89]

I think the above examples are from ledger index 64130309. But I believe they are not created in that ledger but just included as ledger entries.

I was gathering those entries using the "ledger_data" websocket command of the latest, validated ledger.

But you can just get the latest ledger. Almost all RippleState have the "not hex HighNode/LowNode" value and can not be encoded.

[natenichols]

Gotcha, found it. I'm writing up and testing a fix, looks like its a hex parsing issue.

For the time being a ledger_data WebSocket command w/ the binary flag set to true should accomplish essentially the same thing.

Something like whats below gets the encoded info directly from rippled:

{
  "id": "fetches ledger entries as a hex string",
  "ledger_index": "64130309",
  "command": "ledger_data",
  "limit": 10,
  "binary": true
}

Let me know if that does what you're trying to do. Hoping to have a fix out for ripple-binary-codec shortly. Thanks again for opening an issue!

[natenichols]

We have a PR up w/ a fix that successfully encodes the examples above:

You can try it out w/

yarn add ripple-binary-codec@beta 

or:

yarn add ripple-binary-codec@1.1.3-beta.0

[intelliot]

@natenichols is this related to https://github.com/ripple/ripple-binary-codec/pull/121 ?

[natenichols]

@intelliot yeah, I assume so. Which rippled PR made those changes? I can take a look.

Its the same error, and w/ this fix we can remove the edge case for encoding "0".

[intelliot]

@nixer89 Just curious, what's the use case for this? i.e. When/why do you want to ledger_data objects from rippled?

[intelliot]

@natenichols I just wrote a public postmortem and published it under the ripple-lib wiki here: https://github.com/ripple/ripple-lib/wiki/%5BPostmortem%5D-rippled-1.7-encoding-change#root-cause

Hope that is helpful

[nixer89]

@intelliot: I am providing an API which scans a "full ledger" to collect data on XRPL tokens and ledger data analysis (under development).

I use the token API in my own project: https://xumm.community/tokens

This is why I implemented it in the first place (but it is utilized by others already). At the moment it is still growing to provide more data. As example some "full ledger statistics":

But this is an unfinished project yet :)

To calculate the size of a single ledger object I use the ripple-binary-codec to decode/encode the data (I was playing around in receiving data in binary or json, so used decode AND encode). That's when I encountered the above problem.

[nixer89]

@natenichols

I just ran my code again with the version:

"ripple-binary-codec": "^1.1.3-beta.0",

and get the following error:

Error: 19a5 is not a valid hex-string at Function.UInt64.from (..\node_modules\ripple-binary-codec\dist\types\uint-64.js:57:23) at ..\node_modules\ripple-binary-codec\dist\types\st-object.js:129:56 at Array.forEach () at Function.STObject.from (..\node_modules\ripple-binary-codec\dist\types\st-object.js:128:16) at serializeObject (..\node_modules\ripple-binary-codec\dist\binary.js:59:32) at Object.encode (..\node_modules\ripple-binary-codec\dist\index.js:24:12)

I believe the Regex only accepts upper case characters:

var HEX_REGEX = /^[A-F0-9]{1,16}$/;

and the check for "19a5" fails therefore.

The previous examples I have given (when creating the issue), only contained numbers in the hex value, like "283" and no lower case characters.

edit: I have identified the following properties of (mostly RippleState, DirectoryNode, Offers) objects which contain lower case HEX values:

"IndexNext" "IndexPrevious" "LowNode" "HighNode" "ExchangeRate" "OwnerNode"

All other properties of ledger objects, where HEX is stored, are upper case values.

I am wondering if there was a recent rippled change which rendered those values lower case. Because the decode/encode-lib would have had a problem before with lower case values (if the regex above wasn't changed recently)

[natenichols]

Hi @nixer89, nice catch.

Published another beta that adjusts that HEX_REGEX to be:

const HEX_REGEX = /^[a-fA-F0-9]{1,16}$/;

Would you mind giving it a try:

yarn add ripple-binary-codec@1.1.3-beta.1

[nixer89]

Hi @natenichols,

looks good now! I didn't got any error anymore in my tests. Thank you!

Best, Daniel