need command line options to print path and key values of JSON file

[eostermueller]

Describe the Feature Request Need a single command line parameter to display all jsonpaths and accompanying values for all values in a json structure.

Consider this sample json: reservations.json:

{
  "Reservations": [
    {
      "Groups": [],
      "Instances": [
        {
          "ImageId": "ami-a",
          "InstanceId": "i-a",
          "InstanceType": "t2.micro",
          "KeyName": "ubuntu"
        }
      ]
    }
  ]
}

Existing approach: Here is an existing approach (taken from here and not my own work) that displays the jsonpaths/values. Entering the 170-ish char expression is way too verbose/unwieldly, but it achieves the desired results.

cat reservations.json | jq -r 'def kv_to_str($k;$v): "\($k): \($v)"; paths(scalars|true) as $p | kv_to_str("." + ($p|map([.]|tojson)|join("")); getpath($p)|tojson)'
.["Reservations"][0]["Instances"][0]["ImageId"]: "ami-a"
.["Reservations"][0]["Instances"][0]["InstanceId"]: "i-a"
.["Reservations"][0]["Instances"][0]["InstanceType"]: "t2.micro"
.["Reservations"][0]["Instances"][0]["KeyName"]: "ubuntu"

Desired approach: Instead, we need a single jq command line parameter to display the same results, like this if a -p parameter (for paths) were used:

cat reservations.json | jq -p
.["Reservations"][0]["Instances"][0]["ImageId"]: "ami-a"
.["Reservations"][0]["Instances"][0]["InstanceId"]: "i-a"
.["Reservations"][0]["Instances"][0]["InstanceType"]: "t2.micro"
.["Reservations"][0]["Instances"][0]["KeyName"]: "ubuntu"

Formatting the Output

The jsonpath format outputted must be presentable as input. The example above outputted a jsonpath which is reused here successfully as input.

jq '.["Reservations"][0]["Instances"][0]["InstanceType"]' reservations.json
"t2.micro"

With the above solution, the output for larger json or json with more complex object structure is admittedly not picturesque.... the pre-built jsonpaths are so valuable, the below-average formatting doesn't even matter. Perhaps consider one parameter with this poor formatting and a different parameter for better formatting.

Additionally, IDE and other tools could use the pre-built json paths, and this jq feature could provide those paths. jq could create a new IDE-friendly json structure that contains each pre-built json path along with the line/character number of it's value from the original json. Just to show that people would use this kind of thing, here is on existing IDE plugin that generates a jsonpath when the user selects a json value.

Rationale

Pre-built json paths expedite work -- look at the proliferation of online tools that pre-build jsonpaths: https://jsonpathfinder.com/ and https://www.site24x7.com/tools/jsonpath-finder-validator.html.

I would like this functionality available while taking a kubernetes exam. jq is available in the test environment, but I'll have to memorize the 170 characters and hand-key it into my .bashrc test environment....what a pain.

This would be helpful for everyone who does work with kubernetes and other json-heavy administration.

Environment (please complete the following information):

• OS and Version: linux / RedHat 9.2
• jq version 1.6

Additional context

Proper formatting for the jq expression to make it easier on the eyes:

def kv_to_str($k; $v): "\($k): \($v)";

paths(scalars | true) as $p |
    kv_to_str(
        "." + ($p | map([.] | tojson) | join(""));
        getpath($p) | tojson
    )

Just to show that there is interest in this kind of thing, here is a similar question with it's own answers.

[wader]

Hi, interesting idea. I wonder if it could be modelled as another input and output format (#467)? thinking this basically is the gron "format"? example:

$ jq -n '{a:"a",b:[1]}' | gron
json = {};
json.a = "a";
json.b = [];
json.b[0] = 1;

$ jq -n '{a:"a",b:[1]}' | gron | gron -u
{
  "a": "a",
  "b": [
     1
  ]
}

I think there also has been some discussion, that i can't find now, about adding some helpers for dealing paths. That would at least simplify doing things like this a bit, for example fq has path_to_expr and expr_to_path:

$ fq -cn '".a[0]" | expr_to_path | ., path_to_expr'
["a",0]
".a[0]"

[eostermueller]

@wader, didn't know about gron, thanks for sharing. I was thinking this would be much easier. When -p is present, the -r would be implied and the expression mentioned above would be used. Also, I just updated the issue to state that the output jsonpath format should be presentable as input.

[wader]

Agree on output should be in input format, maybe should even be <path> = <value> to be valid jq? not sure what i think about -p yet, have to think about it. It would also be great with input from other jq maintainers on this but it might take a while.

[pkoppstein]

@wader wrote:

It would also be great with input from other jq maintainers

The jq authors and maintainers seem to be in pretty close agreement that new command-line options are strenuously to be avoided, especially when a straightforward jq program can fill the bill.

In this particular case, the existing --stream option does seem to meet the stated functional requirements. In fact, the JSON stream produced by --stream can be used programmatically even more easily that the proposed output, the point being that the output interfaces nicely with jq's setpath.

Please also note even if the output produced by --stream -c is itself deemed unsatisfactory, it can nevertheless very easily and succinctly be tweaked to achieve variants such as the one that has been proposed.

[wader]

@wader wrote:

It would also be great with input from other jq maintainers

The jq authors and maintainers seem to be in pretty close agreement that new command-line options are strenuously to be avoided, especially when a straightforward jq program can fill the bill.

Agreed

In this particular case, the existing --stream option does seem to meet the stated functional requirements. In fact, the JSON stream produced by --stream can be used programmatically even more easily that the proposed output, the point being that the output interfaces nicely with jq's setpath.

Please also note even if the output produced by --stream -c is itself deemed unsatisfactory, it can nevertheless very easily and succinctly be tweaked to achieve variants such as the one that has been proposed.

Good point, and it's way more readable then i expected 👍

$ pbpaste | jq --stream -c 'select(length==2)'
[["Reservations",0,"Groups"],[]]
[["Reservations",0,"Instances",0,"ImageId"],"ami-a"]
[["Reservations",0,"Instances",0,"InstanceId"],"i-a"]
[["Reservations",0,"Instances",0,"InstanceType"],"t2.micro"]
[["Reservations",0,"Instances",0,"KeyName"],"ubuntu"]

[eostermueller]

...that's a step in the right direction, thanks for putting that together.

[mountaineerbr]

I have something that may help in my .bashrc:

#get all paths of a json
jqpath()
{
     jq -r 'def path2text($value):
          def tos: if type == "number" then . else tojson end;
          reduce .[] as $segment ("";  .
            + ($segment
               | if type == "string" then "." + . else "[\(.)]" end))
          + " = \($value | tos)";

        paths(scalars) as $p
          | getpath($p) as $v
          | $p | path2text($v)' "$@"
}
jqpath2()
{
    jq -rc 'path(..)|[.[]|tostring]|join("/")' "$@"
    #jq -r '[path(..)|map(if type=="number" then "[]" else tostring end)|join(".")|split(".[]")|join("[]")]|unique|map("."+.)|.[]' "$@"

}
#https://github.com/stedolan/jq/issues/243
#also see `gron' package to flatten json
#https://www.datafix.com.au/BASHing/2022-03-23.html