partially compact ("collapse"?) JSON output

[kjell]

I'd like to output "mostly pretty" JSON, with overly verbose objects compacted. Something like:

> jq ". | compact(.geometry.coordinates, .other.path.expression)" something.geojson
{
  "type": "Feature",
  "properties": [],
  "geometry": {
    "type": "Polygon",
    "coordinates": [[[0.2944921875, -0.49865941695124], [0.2944921875, -0.48077953413874], [0.49800732421875, -0.48077953413874], [0.49800732421875, -0.49865941695124], [0.2944921875, -0.49865941695124]]]
  }
}

Has this been discussed anywhere?

[ghost]

Would this alter the structure of the JSON object outputted, or just the way it is pretty-printed to the screen?

[kjell]

It would do the exact same thing as --compact-output, but only at the given paths. So it changes the structure of jqs output, whether printed or redirected to a file, but not the value of the JSON.

It reduces 31 lines of pretty-printed JSON to 7 equivalent lines.

[nicowilliams]

A jq function could do this, no doubt, though you'd have to use the raw output option for jq itself...

[ghost]

I see. Well, there's obviously a crucial difference, in that --compact-output is a flag, but your compact proposal is a filter. Personally I see little gain on altering the pretty printer for this purpose.

What @nicowilliams proposes, while feasible, doesn't sound practical, but maybe that's just me.

Could you expand on which is the use case for this feature? Is this a readability concern? What is exactly the problem that is being solved?

How would you feel about a version of --compact-output (let's call it --collapse-output) that skipped the newlines on objects and arrays whose width were below a given constant? As an example, this hypothetical flag would make the output you gave look like this:

jq --collapse-output=80 "." something.geojson
{
  "type": "Feature",
  "properties": [],
  "geometry": {
    "type": "Polygon",
    "coordinates": [
      [
        [0.2944921875, -0.49865941695124],
        [0.2944921875, -0.48077953413874],
        [0.49800732421875, -0.48077953413874],
        [0.49800732421875, -0.49865941695124],
        [0.2944921875, -0.49865941695124]
      ]
    ]
  }
}

[evandrix]

bump

[nicowilliams]

@evandrix What do you propose?

[nicowilliams]

@kjell Why do you want this? Is it to help visualize some JSON? If that's all then.. just delpath the parts you don't want :)

[nicowilliams]

jq is a JSON processor, but it works mainly with parsed values, which is why we have so many issues complaining about changes in formatting of numbers, for example (and before that order of object names/keys). jq is not a very formatter: it has the kinds of options most JSON encoders have (e.g., compact vs. pretty-printed with some number of spaces for indentation), but that's it. Adding a way to specify different formatting options at different parts of a JSON value tree is... not trivial. It could be done, I suppose, but presumably the paths where different options are to be used should be expressed as jq expressions, which would have to be evaluated by... the encoder, which kinda means that either the encoder runs a jq program separate from the main one, or the encoder has to be jq-coded itself. I think it would have to be the latter.

A jq-coded encoder is not infeasible, but as @slapresta says, it's not terribly practical, though mostly because one doesn't exist yet.

[kjell]

I think this makes more sense as a second program.

jq '.' file.json | collapse-json.

It's not so much to visualize the JSON as to make it as human-readable as possible without deleting anything. In my example above, my brain will never understand those lat/longs no matter how they're printed. It's enough to know that it has some coordinates.

But glancing at the JSON I might care about type and properties. Both compacting and pretty printing give more prominence than I want to the coordinates raw data, so the human-readable stuff is a bit harder to grok.

I'm thinking of it like JSON chartjunk, "a lot of ink that does not tell the viewer anything new". So minimizing it would be great, but it's not important. (And sorry for my silence, I lost track of this over the new year.)

[jcracknell]

As an alternative, how about altering the behavior of the flag to compact below a certain depth, where the current behavior is effectively -c0? This way you can produce output with a readable high-level structure that doesn't get too crazy for large documents.

[balta2ar]

@nicowilliams I wish jq could be a little smarter at pretty-printing. For example, here is what underscore-cli pretty-prints:

{
  "1142": {
    "end_datetime": "2017-03-14 23:59",
    "lambda": { "caps": { }, "pacing": "asap" },
    "start_datetime": "2016-03-15 00:00",
    "targeting": {
      "weekly_schedule": {
        "Fri": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Mon": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Sat": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Sun": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Thu": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Tue": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Wed": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
      }
    }
  },

Here is jq output:

{
  "1142": {
    "end_datetime": "2017-03-14 23:59",
    "lambda": {
      "caps": {},
      "pacing": "asap"
    },
    "start_datetime": "2016-03-15 00:00",
    "targeting": {
      "weekly_schedule": {
        "Fri": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23
        ],
        "Mon": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23
        ],
        "Sat": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23
        ],
        "Sun": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23
        ],
        "Thu": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23
        ],
        "Tue": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23
        ],
        "Wed": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23
        ]
      }
    }
  },

[Pysis868]

While looking at kjell's post, I think having it be a filter feels the best, something with the default nature of jq expanding/pretty-printing by default, but some filter like collapse or keep-collapsed in the mix.

Although, probably with what nicowilliams has said, if I'm understanding the tool correctly, filters just operate on the data, and the formatting stage is last and separate from our control. That could change possibly, it would just need some engineering.

Maybe data going through the chain of filters would get 'flagged' with any specific formatting options chosen for them. In the late formatting stage, the 'global' tool flags would be used for all data segments as the iteration process executes, but the specific flags would take precedence if any were found.

The example and idea from balta2ar does also sound good, just not as potentially flexible, and if it is easy enough to implement, maybe leave it open to some configuration at least, if possible.

[nicowilliams]

Assuming a tojson that lets you specify formatting options, your run jq --raw-output ... and you'd manage all the JSON output by using tojson builtins appropriately.

In principle we could remove, or, more likely, reimplement a number of jq command-line options and behaviors by just wrapping the top-level program with jq code that manages standard I/O as expected, with the main() C function always running with input and output in raw mode and letting the jq-coded wrapper do the rest.

Because of that we're reluctant to add new command-line arguments where one can get the same behavior from appropriate use of existing options and jq builtins.

[foragerr]

@balta2ar thanks for pointing me to underscore_cli

[alexshpilkin]

Re how to do and specify this, the Haskell people have long ago solved the general case using backtracking. See Hughes (1995), Wadler (1998), discussion in elixir-lang/elixir#1047. There is also, apparently, an algorithm of Oppen that makes choosing the optimal layout linear-time. Not sure if such generality is actually necessary in this rather limited case, though.

[ghost]

Python pprint works reasonably well:

$ cat wh.py
xr = [
  [
    1428578978,
    "yt_I4Uy3Zjg50o",
    "Camera Obscura - Desire Lines (4AD Session)",
    2013
  ],
  [
    1428578548,
    "yt_199XS8ucYiM",
    "Doe Paoro - Traveling",
    2015
  ],
  [
    1428578095,
    "yt_TBt62fl1ZLM",
    "Trance Soundtrack - Raw Umber",
    2013
  ]
]
import pprint
pprint.pprint(xr, compact=True)

Result:

$ python3 wh.py
[[1428578978, 'yt_I4Uy3Zjg50o', 'Camera Obscura - Desire Lines (4AD Session)',
  2013],
 [1428578548, 'yt_199XS8ucYiM', 'Doe Paoro - Traveling', 2015],
 [1428578095, 'yt_TBt62fl1ZLM', 'Trance Soundtrack - Raw Umber', 2013]]

http://docs.python.org/library/pprint

[yanOnGithub]

My script puts the last level of braces on one line. Feed the output of "jq ." to stdin. I think it can be extended to support the last level of brackets as well.

#!/bin/perl -0

while(<>) {
    my @array = split(/(\{[^{}]+\})/, $_);
    for(my $a = 1; $a < scalar(@array); $a += 2) {
        $array[$a] =~ s!^\s+!!mg;
        $array[$a] =~ s![\r\n]+! !g;
    }
    print join "", @array;
}

[pkoppstein]

underscore (https://github.com/ddopson/underscore-cli) has a parameter that influences the degree of compactness, e.g.

$ underscore --wrapwidth 100 pretty < input.json
{
  context: [
    {
      name: "John",
      node: [{ id: 1, detail: "hello" }, { id: 2, detail: "world" }]
    },
    { name: "Andy", node: [{ id: 3, detail: "andy" }] },
    { name: "Dave", node: [{ id: 4, detail: "dave" }] }
  ]
}

and:

$ underscore --wrapwidth 72 pretty < input.json
{
  context: [
    {
      name: "John",
      node: [
        { id: 1, detail: "hello" },
        { id: 2, detail: "world" }
      ]
    },
    {
      name: "Andy",
      node: [{ id: 3, detail: "andy" }]
    },
    {
      name: "Dave",
      node: [{ id: 4, detail: "dave" }]
    }
  ]
}

[jjlorenzo]

Here is what I'm using for inspecting a json log file

tail -f log.json | while read event; do underscore --wrapwidth `tput cols` pretty -d "$event" ; done

[tomwhoiscontrary]

My script puts the last level of braces on one line.

I think this would be 80% of what i need. For example, it would turn this:

{
  "BXSW": {
    "bid": {
      "price": -0.455,
      "updateTimestamp": "2019-12-02T12:17:13Z"
    },
    "offer": {
      "price": -0.437,
      "updateTimestamp": "2019-12-02T12:17:13Z"
    }
  },
  "CASW": {
    "bid": {
      "price": -0.4486,
      "updateTimestamp": "2019-12-02T10:14:15Z"
    },
    "offer": {
      "price": -0.4434,
      "updateTimestamp": "2019-12-02T10:14:15Z"
    }
  },
  "CMPN": {
    "bid": {
      "price": -0.448,
      "updateTimestamp": "2019-12-02T15:36:11Z"
    },
    "offer": {
      "price": -0.44,
      "updateTimestamp": "2019-12-02T15:36:11Z"
    }
  }
}

Into the substantially more readable:

{
  "BXSW": {
    "bid": { "price": -0.455, "updateTimestamp": "2019-12-02T12:17:13Z" },
    "offer": { "price": -0.437, "updateTimestamp": "2019-12-02T12:17:13Z" }
  },
  "CASW": {
    "bid": { "price": -0.4486, "updateTimestamp": "2019-12-02T10:14:15Z" },
    "offer": { "price": -0.4434, "updateTimestamp": "2019-12-02T10:14:15Z" }
  },
  "CMPN": {
    "bid": { "price": -0.448, "updateTimestamp": "2019-12-02T15:36:11Z" },
    "offer": { "price": -0.44, "updateTimestamp": "2019-12-02T15:36:11Z" }
  }
}

From reading the discussion above, perhaps this does not belong in jq, which is focused on the content of the JSON, but in some separate tool that is specifically about pretty-printing.

[jpbochi]

was there any attempt to implement this in jq?

[deepfire]

Still as relevant as ever..

And I second the idea of having the wrapping width as a parameter.

[lefuturiste]

Same need here but for me a max-printing depth parameter could also be handy. I agree that it could be the job of another program, that would fit nicely in the unix philosophie. I'm looking for a program that could do that, if you may know it I'm interested.

[Jan-Bruun-Andersen]

@yanOnGithub - I used your little JSON compactor as part of a tool I created for compacting some Elasticsearch template files (written in JSON). The tool can be found here: https://github.com/Jan-Bruun-Andersen/elastic-legacy-to-composable

[nngo]

I see. Well, there's obviously a crucial difference, in that --compact-output is a flag, but your compact proposal is a filter. Personally I see little gain on altering the pretty printer for this purpose.

What @nicowilliams proposes, while feasible, doesn't sound practical, but maybe that's just me.

Could you expand on which is the use case for this feature? Is this a readability concern? What is exactly the problem that is being solved?

How would you feel about a version of --compact-output (let's call it --collapse-output) that skipped the newlines on objects and arrays whose width were below a given constant? As an example, this hypothetical flag would make the output you gave look like this:

jq --collapse-output=80 "." something.geojson
{
  "type": "Feature",
  "properties": [],
  "geometry": {
    "type": "Polygon",
    "coordinates": [
      [
        [0.2944921875, -0.49865941695124],
        [0.2944921875, -0.48077953413874],
        [0.49800732421875, -0.48077953413874],
        [0.49800732421875, -0.49865941695124],
        [0.2944921875, -0.49865941695124]
      ]
    ]
  }
}

I like this idea of --collapse-output option and it's output, Also, like the output of https://github.com/ddopson/underscore-cli for human readablity, especially with arrays of numbers as pointed out in this comment https://github.com/stedolan/jq/issues/643#issuecomment-284646767 and example

{
  "1142": {
    "end_datetime": "2017-03-14 23:59",
    "lambda": { "caps": { }, "pacing": "asap" },
    "start_datetime": "2016-03-15 00:00",
    "targeting": {
      "weekly_schedule": {
        "Fri": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Mon": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Sat": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Sun": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Thu": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Tue": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
        "Wed": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
      }
    }
  },

or similar to https://github.com/j-brooke/FracturedJson output

{
    "SimpleArray": [
          2,   3,   5,   7,  11,  13,  17,  19,  23,  29,  31,  37,  41,  43,  47,  53,  59,  61,
         67,  71,  73,  79,  83,  89,  97, 101, 103, 107, 109, 113
    ],
    "ObjectColumnsArrayRows": {
        "Katherine": ["blue"      , "lightblue", "black"       ],
        "Logan"    : ["yellow"    , "blue"     , "black", "red"],
        "Erik"     : ["red"       , "purple"                   ],
        "Jean"     : ["lightgreen", "yellow"   , "black"       ]
    },
    "ArrayColumnsObjectRows": [
        { "type": "turret"   , "hp": 400, "loc": {"x": 47, "y":  -4}, "flags": "S"   },
        { "type": "assassin" , "hp":  80, "loc": {"x": 12, "y":   6}, "flags": "Q"   },
        { "type": "berserker", "hp": 150, "loc": {"x":  0, "y":   0}                 },
        { "type": "pittrap"  ,            "loc": {"x": 10, "y": -14}, "flags": "S,I" }
    ],
    "ComplexArray": [
        [19,  2],
        [ 3,  8],
        [14,  0],
        [ 9,  9],
        [ 9,  9],
        [ 0,  3],
        [10,  1],
        [ 9,  1],
        [ 9,  2],
        [ 6, 13],
        [18,  5],
        [ 4, 11],
        [12,  2]
    ]
}

[jregehr]

bump

[emanuele6]

Someone asked about this on IRC a while ago.

A while even earlier, I happened to have written this jq function that given a value, it formats it to JSON with indendentation. Here is an example script that uses it (use it as ./toprettyjson.jq N; effectively works like jq --indent N ., but it works as you would expect if N is 0 or greater than 7:

#!/bin/sh --
# \
exec jq -f "$0" -r --args -- "$@"

def toprettyjson($n):
  def _toprettyjson($l):
    ($l * $n * " " // "") as $indent |
    if (isempty(scalars) | not) or IN({}, []) then
      tojson
    elif type == "object" then
      "{\n" + (
        to_entries |
        map($indent + ($n * " " // "") +
          "\(.key | tojson): \(.value | _toprettyjson($l + 1))"
        ) |
        join(",\n")
      ) + "\n" + $indent + "}"
    else # array
      "[\n" + (
        map($indent + ($n * " " // "") +
            _toprettyjson($l + 1)) |
        join(",\n")
      ) + "\n" + $indent + "]"
    end;
  _toprettyjson(0);

toprettyjson($ARGS.positional[0] | tonumber? // 4)

---

They wanted to "compact" .windows[].tabs[] | objects, so I told them they could adapt my code to do that like so:

#!/bin/sh --
# \
exec jq -f "$0" -r --args -- "$@"

def toprettyjson($n):
  def _toprettyjson($l; $path):
    ($l * $n * " " // "") as $indent |
    if (isempty(scalars) | not) or IN({}, []) then
      tojson
    elif type == "object" then
      if $path == [ "windows", 0, "tabs", 0 ] then
        tojson
      else
        "{\n" + (
          to_entries |
          map($indent + ($n * " " // "") +
            "\(.key | tojson): \(
              .key as $k |
              .value | _toprettyjson($l + 1; $path + [ $k ]))"
          ) |
          join(",\n")
        ) + "\n" + $indent + "}"
      end
    else # array
      "[\n" + (
        map($indent + ($n * " " // "") +
            _toprettyjson($l + 1; $path + [0])) |
        join(",\n")
      ) + "\n" + $indent + "]"
    end;
  _toprettyjson(0; []);

toprettyjson($ARGS.positional[0] | tonumber? // 4)

Example:

bash-5.1$ # example input
bash-5.1$ printf %s\\n hi hello xyz abc hi | jq -Rn '{windows:[{tabs:[1,2,3,4|{meta:.,foo:input}]}],hi:"hello"}'
{
  "windows": [
    {
      "tabs": [
        {
          "meta": 1,
          "foo": "hi"
        },
        {
          "meta": 2,
          "foo": "hello"
        },
        {
          "meta": 3,
          "foo": "xyz"
        },
        {
          "meta": 4,
          "foo": "abc"
        }
      ]
    }
  ],
  "hi": "hello"
}
bash-5.1$ printf %s\\n hi hello xyz abc hi | jq -Rn '{windows:[{tabs:[1,2,3,4|{meta:.,foo:input}]}],hi:"hello"}' | ./patchedtoprettyjson.jq 2
{
  "windows": [
    {
      "tabs": [
        {"meta":1,"foo":"hi"},
        {"meta":2,"foo":"hello"},
        {"meta":3,"foo":"xyz"},
        {"meta":4,"foo":"abc"}
      ]
    }
  ],
  "hi": "hello"
}

[nicowilliams]

Yes, it would be very nice to have this as a native feature in jq.

[emanuele6]

@nicowilliams You could easily implement that adapting my code as...

def tocompactjson(p; $n):
  [ canonicalize_path(path(p)) ] as $paths |
  def _tocompactjson($l; $path):
    ($l * $n * " " // "") as $indent |
    if IN($paths[]; $path) or (isempty(scalars)|not) or IN({}, []) then
      tojson
    elif type == "object" then
      "{\n" + (
        to_entries |
        map($indent + ($n * " " // "") +
          "\(.key | tojson): \(
            .key as $k |
            .value | _tocompactjson($l + 1; $path + [ $k ]))"
        ) |
        join(",\n")
      ) + "\n" + $indent + "}"
    else # array
      "[\n" + (
        [
          range(length) as $i |
          .[$i] |
          $indent + ($n * " " // "") +
          _tocompactjson($l + 1; $path + [$i])
        ] |
        join(",\n")
      ) + "\n" + $indent + "]"
    end;
  _tocompactjson(0; []);

if canonicalize_path/1 existed...

---

Proof of concept with idenitity canonicalize_path/1. It won't work if you use a negative index (.foo[-1]) or slice (.bar[1:3][]) in the path directly or indirectly:

$ jq -r 'def canonicalize_path(x): x; def tocompactjson(p; $n):
  [ canonicalize_path(path(p)) ] as $paths |
  def _tocompactjson($l; $path):
    ($l * $n * " " // "") as $indent |
    if IN($paths[]; $path) or (isempty(scalars)|not) or IN({}, []) then
      tojson
    elif type == "object" then
      "{\n" + (
        to_entries |
        map($indent + ($n * " " // "") +
          "\(.key | tojson): \(
            .key as $k |
            .value | _tocompactjson($l + 1; $path + [ $k ]))"
        ) |
        join(",\n")
      ) + "\n" + $indent + "}"
    else # array
      "[\n" + (
        [
          range(length) as $i |
          .[$i] |
          $indent + ($n * " " // "") +
          _tocompactjson($l + 1; $path + [$i])
        ] |
        join(",\n")
      ) + "\n" + $indent + "]"
    end;
  _tocompactjson(0; []);
tocompactjson(.foo, .baz[2]; 2)
' <<< '
{"foo":[{"1":2,"3":null},1],"baz":[{"1":2,"3":null},1,{"1":2,"3":null},3],"bar":[{"1":2,"3":null},1]}
'
{
  "foo": [{"1":2,"3":null},1],
  "baz": [
    {
      "1": 2,
      "3": null
    },
    1,
    {"1":2,"3":null},
    3
  ],
  "bar": [
    {
      "1": 2,
      "3": null
    },
    1
  ]
}

[av930]

any updates?