mikefarah / yq

yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor
https://mikefarah.gitbook.io/yq/
MIT License
12.36k stars 602 forks source link

Json to Yaml Mangling Large Integer #2122

Open MatthewChau opened 3 months ago

MatthewChau commented 3 months ago

When converting from json to yaml, a large integer is getting converted to scientific notation. This leads to problems with rounding.

Version of yq: 4.44.2 Operating system: linux_amd64 (running in WSL) Installed via: Binary release

Input json

"schema": {
    "maximum": 9223372036854775807,
    "minimum": 1,
    "type": "integer",
    "format": "int64"
}

Command

./yq_linux_amd64 -P -oy test.json

Actual behavior

schema:
    maximum: 9.223372036854776e+18
    minimum: 1
    type: integer
    format: int64

Expected behavior

schema:
    maximum: 9223372036854775807
    minimum: 1
    type: integer
    format: int64

Additional context Expected behavior matches actual behavior when testing with v4.14.1

./yq_linux_amd64 eval -P -oy test.json
dhduvall commented 3 months ago

Even worse (IMO):

❯ echo "3681579493102783500" | yq --input-format json
3681579493102783488

Actually, all numbers from 3681579493102783232 to 3681579493102783744 convert to 3681579493102783488. The next set of numbers lower convert to 3681579493102782976.

MatthewChau commented 2 months ago

Is this less of a bug and more of a behavior change? Although I see that this issue has been brought up before in older versions. https://github.com/mikefarah/yq/issues/245 Did some additional testing and the change happens with v4.32.1.

./yq_linux_amd64_v4.31.2 -oy test.json

Output

"schema": {"maximum": 9223372036854775807, "minimum": 1, "type": "integer", "format": "int64"}
./yq_linux_amd64_v4.32.1 -oy test.json

Output

schema
---
maximum: 9.223372036854776e+18
minimum: 1
type: integer
format: int64
./yq_linux_amd64_v4.32.1 -oj test.json

Output

"schema"
{
    "maximum": 9223372036854776000,
    "minimum": 1,
    "type": "integer",
    "format": "int64"
}
lrh3321 commented 1 month ago

https://www.rfc-editor.org/rfc/rfc8259#section-6

Note that when such software is used, numbers that are integers and are in the range $[-(2^{53})+1, (2^{53})-1]$ are interoperable in the sense that implementations will agree exactly on their numeric values.