We need a policy for how to deal with requests for extended JSON features

[nicowilliams]

Options:

• the old Postel principle This is problematic because we would end up having every extension always enabled for parsing.
• require command-line options to enable JSON extensions / permissiveness This is problematic because we could easily end up with too many options, so we'd probably have a single option that has an argument, but even so we've been wanting to add no more options.
• have multiple input formats that can be selected with a command-line option
• have multiple input formats that can be selected with jq code using raw input / raw output using fromjson-/tojson-like builtins

The last two are nicer because it's a cleaner path to adding support for reading YAML and other things, and with the last one then users can just jq -Rr 'from_whatever(inputs) | ... to_whatever' (with from_whatever(s) being something like def from_whatever(s): foreach s as $i (null; from_whatever_incremental; select(.ready).value) | ... | to_whatever').

[nicowilliams]

We really should have support for:

• various JSON "extensions"
• or really, just JSON5
• CBOR and other binary JSON formats (e.g., PostgreSQL's JSONB)
• YAML
• XML
• TOML
• GRON
• and so on

[nicowilliams]

From the SQLite3 JSON docs:

[nicowilliams]

My strong preference is that we add no command-line options for input/output formats -- just require the use of -R and -r, and provide suitable builtins for all sorts of input and output formats.

[pkoppstein]

@nicowilliams wrote:

My strong preference is that we add no command-line options for input/output formats

That would be fine except for the question of "legacy" extensions (00 for 0, etc). Other than ignoring the issue, there are several possibilities, notably:

1) jq 2.0 should simply disallow them (except as determined by new filters); 2) there should be a new --strict option (for the sake of backwards-compatibility), as per https://github.com/jqlang/jq/issues/2643

[nicowilliams]

My strong preference is that we add no command-line options for input/output formats

That would be fine except for the question of "legacy" extensions

Indeed, there's no question here of breaking backwards compatibility, only of what to do with requests for and PRs that add new extensions.

[wader]

from_whatever/to_whatever is how fq does at the moment. It can also autodetect some input formats to cut down on typing:

$ echo '<a>123</a>' | fq -r '., to_yaml, to_jq, tojson({indent: 8}), to_xml({indent: 2})'
{
  "a": "123"
}
a: "123"

{a:"123"}
{
        "a": "123"
}
<a>123</a>

(also shows one of annoying thing, should to_whatever produce a string with ending newline for some formats like yaml? 🤔)

Feel natural but i suspect it's hard to understand/discover by someone not familiar with jq the language. Someone new to jq who wants to convert i think will look for --to/--from CLI arguments.

[flipmcf]

We really should have support for:

• YAML
• XML
• TOML
• GRON
• and so on

NO.

I'm sorry, I have only recently found jq and found myself reading the issues, and just couldn't stop myself from commenting on this.

today, jq this is a spectacular command line utility for json and you will only make it worse by adding this functionality. There are perfectly good XML, YAML and other parsers (libxml2) and perfectly good languages for processing them (xpat, XSLT) and perfectly good command line tools (xmlstarlet, yq)

Stay focused on JSON and JSON extentions.

Don't break 'singlemindedness'. Being able to print out XML or YAML from a json input sounds intriguing, but certainly not input.

yq is taking Json input, so let them be the 'one ring'. do Json, and do it more awesome than them.

sed, awk, and grep aren't going anywhere either.

Thanks for listening to a stranger.

[nicowilliams]

@flipmcf thanks for your feedback. If and when we add support for more formats, you won't have to use them if you don't want to. It certainly makes sense for jq to support alternative encodings of JSON (like CBOR); the others might not be natural to use jq with (though I believe we can make it natural enough).

[pkoppstein]

@flipmcf - Yes, bloat causes significant costs in multiple respects (not least the problems associated with complexity in general and feature interaction in particular), especially for volunteer projects such as this one, for which maintenance has already proven to be a major issue.

Since feature enhancements are what jq volunteers typically like to work on, perhaps it would help if you could identify some that you think should have high priority in the near term, say for jq 1.8.

[nicowilliams]

SQLite3 has lots of options for excluding functionality to get bloat down. If bloat becomes an issue then we should do something similar.