search

jqlang / jq

Command-line JSON processor
https://jqlang.github.io/jq/
Other
30.38k stars 1.58k forks source link

Seg fault when using match or catpure on lines longer than 170 #1288

Closed doitian closed 1 year ago

doitian commented 7 years ago

jq 1.5 on Mac installed through Homebrew.

The test script

#!/bin/bash

rep() {
  local str="$1"
  local -i count="$2"
  while [ "$count" -gt 0 ]; do
    echo -n "$str"
    count=$(( count - 1 ))
  done
}

log() {
cat <<LOG
[2345678] $(rep '1234567890' 17)
LOG
}

uname -a
jq --version
log | jq -R 'match("([^ ]+) (.+)")'

output:

Darwin ian-rmbp.local 16.1.0 Darwin Kernel Version 16.1.0: Thu Oct 13 21:26:57 PDT 2016; root:xnu-3789.21.3~60/RELEASE_X86_64 x86_64
jq-1.5
/private/tmp/test.sh: line 20: 93111 Done                    log
     93112 Segmentation fault: 11  | jq -R 'match("([^ ]+) (.+)")'

shell returned 139

If I change the number 17 in test script to 16, it works.

pkoppstein commented 7 years ago

@doitian - I had no trouble running your script on a Mac, whether using jq 1.5 or the current "master" version. Here are extracts from the output with the second argument of rep() set to 18 for good measure:

Darwin Mac-mini.local 13.4.0 Darwin Kernel Version 13.4.0: Mon Jan 11 18:17:34 PST 2016; root:xnu-2422.115.15~1/RELEASE_X86_64 x86_64
jq-1.5
{
  "offset": 0,
  "length": 190,
  "string": "[2345678] 123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
  "captures": [
    {
      "offset": 0,
      "length": 9,
      "string": "[2345678]",
      "name": null
    },
    {
      "offset": 10,
      "length": 180,
      "string": "123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
      "name": null
    }
  ]
}
doitian commented 7 years ago

The prebuilt binary downloaded here works. Thus a bug of the Homebrew bottle.

doitian commented 7 years ago

The mac 64 binary download from release works. But if I download the jq-1.5.tar.gz and build from source, the built binary segment fault on long lines.

configure

./configure --disable-maintainer-mode
checking for a BSD-compatible install... /usr/local/bin/ginstall -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/local/bin/gmkdir -p
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for ar... ar
checking the archiver (ar) interface... ar
checking whether to enable maintainer-specific portions of Makefiles... no
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) gcc3
checking for gcc option to accept ISO C99... none needed
checking for gcc option to accept ISO Standard C... (cached) none needed
checking how to run the C preprocessor... gcc -E
checking for bison... bison -y
checking build system type... x86_64-apple-darwin16.1.0
checking host system type... x86_64-apple-darwin16.1.0
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
checking if the linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 196608
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-apple-darwin16.1.0 file names to x86_64-apple-darwin16.1.0 format... func_convert_file_noop
checking how to convert x86_64-apple-darwin16.1.0 file names to toolchain format... func_convert_file_noop
checking for /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for archiver @FILE support... no
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for mt... no
checking if : is a manifest tool... no
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... yes
checking for -exported_symbols_list linker flag... yes
checking for -force_load linker flag... yes
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin16.1.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking oniguruma.h usability... yes
checking oniguruma.h presence... yes
checking for oniguruma.h... yes
checking for onig_version in -lonig... yes
checking for valgrind... no
configure: WARNING: valgrind is required to test jq.
checking for memmem... yes
checking for mkstemp... yes
checking for bundle... bundle
checking for Ruby dependencies... configure: WARNING: The following gems are missing
 * minitest (5.8.0)
 * activesupport (4.2.3)
 * maruku (0.7.2)
 * sass (3.4.16)
 * tilt (2.0.1)
 * sinatra (1.4.6)
 * watch (0.1.0)
 * yui-compressor (0.12.0)
 * bonsai (1.4.9)
 * hpricot (0.8.6)
 * mustache (0.99.8)
 * ronn (0.7.3)
Install missing gems with `bundle install`
*****************************************************************
*  Ruby dependencies for building jq documentation not found.   *
*  You can still build, install and hack on jq, but the manpage *
*  will not be rebuilt and some of the tests won't run.         *
*  See docs/README.md for how to install the docs dependencies. *
*****************************************************************
no
checking for size_t... yes
checking for working alloca.h... yes
checking for alloca... yes
checking for isatty... yes
checking for _isatty... no
checking for strptime... yes
checking for strftime... yes
checking for timegm... yes
checking for gmtime_r... yes
checking for gmtime... yes
checking for gettimeofday... yes
checking for struct tm.tm_gmtoff... yes
checking for struct tm.__tm_gmtoff... no
checking for acos... yes
checking for acosh... yes
checking for asin... yes
checking for asinh... yes
checking for atan... yes
checking for atanh... yes
checking for cbrt... yes
checking for cos... yes
checking for cosh... yes
checking for exp2... yes
checking for exp... yes
checking for floor... yes
checking for j0... yes
checking for j1... yes
checking for log10... yes
checking for log2... yes
checking for log... yes
checking for sin... yes
checking for sinh... yes
checking for sqrt... yes
checking for tan... yes
checking for tanh... yes
checking for tgamma... yes
checking for y0... yes
checking for y1... yes
checking for pow... yes
checking for atan2... yes
checking for hypot... yes
checking for remainder... yes
checking for thread-local storage... yes
checking whether byte ordering is bigendian... no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: executing depfiles commands
config.status: executing libtool commands

Core dump

(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff8f3fd15e libsystem_platform.dylib`_platform_memmove$VARIANT$Nehalem + 254, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff8f3fd15e libsystem_platform.dylib`_platform_memmove$VARIANT$Nehalem + 254
    frame #1: 0x0000000103548f55 libonig.4.dylib`stack_double + 141
    frame #2: 0x0000000103543a57 libonig.4.dylib`match_at + 3294
    frame #3: 0x0000000103548029 libonig.4.dylib`onig_search + 2265
    frame #4: 0x00000001034f3fca jq`f_match(jq=<unavailable>, input=<unavailable>, regex=<unavailable>, modifiers=jv @ 0x00007fff5c71f260, testmode=jv @ 0x00007fff5c71f270) + 1626 at builtin.c:698 [opt]
    frame #5: 0x00000001034edc33 jq`jq_next(jq=<unavailable>) + 7427 at execute.c:785 [opt]
    frame #6: 0x00000001034e2e35 jq`process(jq=0x00007faf77c028e0, value=<unavailable>, flags=<unavailable>, dumpopts=645) + 37 at main.c:125 [opt]
    frame #7: 0x00000001034e2935 jq`main(argc=<unavailable>, argv=<unavailable>) + 4693 at main.c:530 [opt]
    frame #8: 0x00007fff8f1f0255 libdyld.dylib`start + 1
    frame #9: 0x00007fff8f1f0255 libdyld.dylib`start + 1

I think Homebrew build environment and mine is different from jq release build machine.