01mf02 / jaq

A jq clone focussed on correctness, speed, and simplicity
MIT License
2.7k stars 67 forks source link

Panic on huge CVE JSON feed #146

Closed franchb closed 8 months ago

franchb commented 8 months ago

Hi!

Thank you for this interesting tool. I experiment with jaq occasionally. Today I caught a panic on some huge JSON feed.

Steps for reproduce:

  1. Download sample huge JSON file of CVE:
curl https://kazepublic.blob.core.windows.net/cvefree/data.json --output data.json
  1. Pipe this file to jaq:
cat data.json | jaq
cat data.json | jaq
thread 'main' panicked at /home/iru/.cargo/registry/src/index.crates.io-6f17d22bba15001f/jaq-interpret-1.2.0/src/filter.rs:227:16:
index out of bounds: the len is 0 but the index is 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

If I pipe the same file in jq:

cat data.json | jaq

jq parses this JSON feed and prints the output:

    {
      "cve": "CVE-2003-0002",
      "last_modified_datetime": "2018-10-12T21:32:00.000Z",
      "published_datetime": "2003-02-07T05:00:00.000Z",
      "cvssv2": 6.8,
      "cvssv3": null,
      "epss": "0.140860000",
      "cti_count": null,
      "social_media_audience": null,
      "vendors": [
        "microsoft"
      ],
      "software_cpes": [
        "cpe:2.3:a:microsoft:content_management_server:2001:*:*:*:*:*:*:*",
        "cpe:2.3:a:microsoft:content_management_server:2001:sp1:*:*:*:*:*:*"
      ],
      "v_score": 0.39455917,
...
01mf02 commented 8 months ago

Hi @franchb, thanks for this bug report!

This seems to be another bug related to using jaq without input filter. For now, you can just use jaq '.' instead of jaq to make your use case work. Oh, and if you want better file loading performance, use jaq '.' data.json. That way, your file gets loaded via memory mapping, which is faster than reading from a pipe.

01mf02 commented 8 months ago

This is corrected now. Thanks again!