search

01mf02 / jaq

A jq clone focussed on correctness, speed, and simplicity
MIT License
2.76k stars 75 forks source link
jq json query rust

readme

jaq

Build status Crates.io Documentation Rust 1.64+

jaq (pronounced /ʒaːk/, like Jacques[^jacques]) is a clone of the JSON data processing tool jq. jaq aims to support a large subset of jq's syntax and operations.

You can try jaq online on the jaq playground. Instructions for the playground can be found here.

jaq focuses on three goals:

I drew inspiration from another Rust program, namely jql. However, unlike jql, jaq aims to closely imitate jq's syntax and semantics. This should allow users proficient in jq to easily use jaq.

[^jacques]: I wanted to create a tool that should be discreet and obliging, like a good waiter. And when I think of a typical name for a (French) waiter, to my mind comes "Jacques". Later, I found out about the old French word jacquet, meaning "squirrel", which makes for a nice ex post inspiration for the name.

Installation

Binaries

You can download binaries for Linux, Mac, and Windows on the releases page.

You may also install jaq using homebrew on macOS or Linux:

$ brew install jaq
$ brew install --HEAD jaq # latest development version

Or using scoop on Windows:

$ scoop install main/jaq

From Source

To compile jaq, you need a Rust toolchain. See https://rustup.rs/ for instructions. (Note that Rust compilers shipped with Linux distributions may be too outdated to compile jaq.)

Any of the following commands install jaq:

$ cargo install --locked jaq
$ cargo install --locked --git https://github.com/01mf02/jaq # latest development version

On my system, both commands place the executable at ~/.cargo/bin/jaq.

If you have cloned this repository, you can also build jaq by executing one of the commands in the cloned repository:

$ cargo build --release # places binary into target/release/jaq
$ cargo install --locked --path jaq # installs binary

jaq should work on any system supported by Rust. If it does not, please file an issue.

Examples

The following examples should give an impression of what jaq can currently do. You should obtain the same outputs by replacing jaq with jq. If not, your filing an issue would be appreciated. :) The syntax is documented in the jq manual.

Access a field:

$ echo '{"a": 1, "b": 2}' | jaq '.a'
1

Add values:

$ echo '{"a": 1, "b": 2}' | jaq 'add'
3

Construct an array from an object in two ways and show that they are equal:

$ echo '{"a": 1, "b": 2}' | jaq '[.a, .b] == [.[]]'
true

Apply a filter to all elements of an array and filter the results:

$ echo '[0, 1, 2, 3]' | jaq 'map(.*2) | [.[] | select(. < 5)]'
[0, 2, 4]

Read (slurp) input values into an array and get the average of its elements:

$ echo '1 2 3 4' | jaq -s 'add / length'
2.5

Repeatedly apply a filter to itself and output the intermediate results:

$ echo '0' | jaq '[recurse(.+1; . < 3)]'
[0, 1, 2]

Lazily fold over inputs and output intermediate results:

$ seq 1000 | jaq -n 'foreach inputs as $x (0; . + $x)'
1 3 6 10 15 [...]

Performance

The following evaluation consists of several benchmarks that allow comparing the performance of jaq, jq, and gojq. The empty benchmark runs n times the filter empty with null input, serving to measure the startup time. The bf-fib benchmark runs a Brainfuck interpreter written in jq, interpreting a Brainfuck script that produces n Fibonacci numbers. The other benchmarks evaluate various filters with n as input; see bench.sh for details.

I generated the benchmark data with bench.sh target/release/jaq jq-1.7.1 gojq-0.12.16 | tee bench.json on a Linux system with an AMD Ryzen 5 5500U.[^binaries] I then processed the results with a "one-liner" (stretching the term and the line a bit):

jq -rs '.[] | "|`\(.name)`|\(.n)|" + ([.time[] | min | (.*1000|round)? // "N/A"] | min as $total_min | map(if . == $total_min then "**\(.)**" else "\(.)" end) | join("|"))' bench.json

(Of course, you can also use jaq here instead of jq.) Finally, I concatenated the table header with the output and piped it through pandoc -t gfm.

[^binaries]: The binaries for jq-1.7.1 and gojq-0.12.16 were retrieved from their GitHub release pages.

Table: Evaluation results in milliseconds ("N/A" if error or more than 10 seconds).

Benchmark n jaq-2.0 jq-1.7.1 gojq-0.12.16
empty 512 350 540 250
bf-fib 13 480 1260 560
defs 100000 60 N/A 1030
upto 8192 0 480 450
reduce-update 16384 10 550 1380
reverse 1048576 40 690 270
sort 1048576 110 540 560
group-by 1048576 510 1930 1560
min-max 1048576 210 320 250
add 1048576 470 630 1270
kv 131072 110 150 220
kv-update 131072 130 540 480
kv-entries 131072 570 1160 710
ex-implode 1048576 510 1110 600
reduce 1048576 750 890 N/A
try-catch 1048576 290 320 360
repeat 1048576 150 840 530
from 1048576 290 1010 590
pyramid 524288 340 360 510
tree-contains 23 70 610 210
tree-flatten 17 770 370 0
tree-update 17 700 980 1350
tree-paths 17 430 270 870
to-fromjson 65536 40 370 110
ack 7 540 710 1240
range-prop 128 360 310 230
cumsum 1048576 280 380 460
cumsum-xy 1048576 460 480 710

The results show that jaq-2.0 is fastest on 24 benchmarks, whereas jq-1.7.1 is fastest on 1 benchmark and gojq-0.12.16 is fastest on 3 benchmarks. gojq is much faster on tree-flatten because it implements the filter flatten natively instead of by definition.

Features

Here is an overview that summarises:

Contributions to extend jaq are highly welcome.

Basics

Paths

Operators

Definitions

Core filters

Standard filters

These filters are defined via more basic filters. Their definitions are at std.jq.

Numeric filters

jaq imports many filters from libm and follows their type signature.

Full list of numeric filters defined in jaq Zero-argument filters: - [x] `acos` - [x] `acosh` - [x] `asin` - [x] `asinh` - [x] `atan` - [x] `atanh` - [x] `cbrt` - [x] `cos` - [x] `cosh` - [x] `erf` - [x] `erfc` - [x] `exp` - [x] `exp10` - [x] `exp2` - [x] `expm1` - [x] `fabs` - [x] `frexp`, which returns pairs of (float, integer). - [x] `gamma` - [x] `ilogb`, which returns integers. - [x] `j0` - [x] `j1` - [x] `lgamma` - [x] `log` - [x] `log10` - [x] `log1p` - [x] `log2` - [x] `logb` - [x] `modf`, which returns pairs of (float, float). - [x] `nearbyint` - [x] `pow10` - [x] `rint` - [x] `significand` - [x] `sin` - [x] `sinh` - [x] `sqrt` - [x] `tan` - [x] `tanh` - [x] `tgamma` - [x] `trunc` - [x] `y0` - [x] `y1` Two-argument filters that ignore `.`: - [x] `atan2` - [x] `copysign` - [x] `drem` - [x] `fdim` - [x] `fmax` - [x] `fmin` - [x] `fmod` - [x] `hypot` - [x] `jn`, which takes an integer as first argument. - [x] `ldexp`, which takes an integer as second argument. - [x] `nextafter` - [x] `nexttoward` - [x] `pow` - [x] `remainder` - [x] `scalb` - [x] `scalbln`, which takes as integer as second argument. - [x] `yn`, which takes an integer as first argument. Three-argument filters that ignore `.`: - [x] `fma`

Modules

Advanced features

jaq currently does not aim to support several features of jq, such as:

Differences between jq and jaq

Numbers

jq uses 64-bit floating-point numbers (floats) for any number. By contrast, jaq interprets numbers such as 0 or -42 as machine-sized integers and numbers such as 0.0 or 3e8 as 64-bit floats. Many operations in jaq, such as array indexing, check whether the passed numbers are indeed integer. The motivation behind this is to avoid rounding errors that may silently lead to wrong results. For example:

$ jq  -n '[0, 1, 2] | .[1.0000000000000001]'
1
$ jaq -n '[0, 1, 2] | .[1.0000000000000001]'
Error: cannot use 1.0 as integer
$ jaq -n '[0, 1, 2] | .[1]'
1

The rules of jaq are:

Examples:

$ jaq -n '1 + 2'
3
$ jaq -n '10 / 2'
5.0
$ jaq -n '1.0 + 2'
3.0

You can convert an integer to a floating-point number e.g. by adding 0.0, by multiplying with 1.0, or by dividing with 1. You can convert a floating-point number to an integer by round, floor, or ceil:

$ jaq -n '1.2 | [floor, round, ceil]'
[1, 1, 2]

NaN and infinity

In jq, division by 0 yields an error, whereas in jaq, n / 0 yields nan if n == 0, infinite if n > 0, and -infinite if n < 0. jaq's behaviour is closer to the IEEE standard for floating-point arithmetic (IEEE 754).

jaq implements a total ordering on floating-point numbers to allow sorting values. Therefore, it unfortunately has to enforce that nan == nan. (jq gets around this by enforcing that nan < nan is true, yet nan > nan is false, which breaks basic laws about total orders.)

Like jq, jaq prints nan and infinite as null in JSON, because JSON does not support encoding these values as numbers.

Assignments

Like jq, jaq allows for assignments of the form p |= f. However, jaq interprets these assignments differently. Fortunately, in most cases, the result is the same.

In jq, an assignment p |= f first constructs paths to all values that match p. Only then, it applies the filter f to these values.

In jaq, an assignment p |= f applies f immediately to any value matching p. Unlike in jq, assignment does not explicitly construct paths.

jaq's implementation of assignment likely yields higher performance, because it does not construct paths. Furthermore, this allows jaq to use multiple outputs of the right-hand side, whereas jq uses only the first. For example, 0 | (., .) |= (., .+1) yields 0 1 1 2 in jaq, whereas it yields only 0 in jq. However, {a: 1} | .a |= (2, 3) yields {"a": 2} in both jaq and jq, because an object can only associate a single value with any given key, so we cannot use multiple outputs in a meaningful way here.

Because jaq does not construct paths, it does not allow some filters on the left-hand side of assignments, for example first, last, limit: For example, [1, 2, 3] | first(.[]) |= .-1 yields [0, 2, 3] in jq, but is invalid in jaq. Similarly, [1, 2, 3] | limit(2; .[]) |= .-1 yields [0, 1, 3] in jq, but is invalid in jaq. (Inconsequentially, jq also does not allow for last.)

Definitions

Like jq, jaq allows for the definition of filters, such as:

def map(f): [.[] | f];

Arguments can also be passed by value, such as:

def cartesian($f; $g): [$f, $g];

Filter definitions can be nested and recursive, i.e. refer to themselves. That is, a filter such as recurse can be defined in jaq:

def recurse(f): def r: ., (f | r); r;

Since jaq 1.2, jaq optimises tail calls, like jq. Since jaq 1.1, recursive filters can also have non-variable arguments, like in jq. For example:

def f(a): a, f(1+a);

Recursive filters with non-variable arguments can yield surprising effects; for example, a call f(0) builds up calls of the shape f(1+(..(1+0)...)), which leads to exponential execution times.

Recursive filters with non-variable arguments can very frequently be alternatively implemented by either:

All of these options are supported by jaq.

Arguments

Like jq, jaq allows to define arguments via the command line, in particular by the options --arg, --rawfile, --slurpfile. This binds variables to values, and for every variable $x bound to v this way, $ARGS.named contains an entry with key x and value v. For example:

$ jaq -n --arg x 1 --arg y 2 '$x, $y, $ARGS.named'
"1"
"2"
{
  "x": "1",
  "y": "2"
}

Folding

jq and jaq provide filters reduce xs as $x (init; update), foreach xs as $x (init; update), and foreach xs as $x (init; update; project), where foreach xs as $x (init; update) is equivalent to foreach xs as $x (init; update; .).

In jaq, the output of these filters is defined very simply: Assuming that xs evaluates to x0, x1, ..., xn, reduce xs as $x (init; update) evaluates to

init
| x0 as $x | update
| ...
| xn as $x | update

and foreach xs as $x (init; update; project) evaluates to

init |
( x0 as $x | update | project,
( ...
( xn as $x | update | project,
( empty )...)

The interpretation of reduce/foreach in jaq has the following advantages over jq:

Miscellaneous

Contributing

Contributions to jaq are welcome. Please make sure that after your change, cargo test runs successfully.

Acknowledgements

This project was funded through the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101069594.

jaq has also profited from: