search

getsentry / symbolic

Stack trace symbolication library written in Rust
https://github.com/getsentry/symbolic#readme
MIT License
446 stars 75 forks source link

Index out of range panic at src/libcore/slice/mod.rs:2413 #159

Closed AdrianLC closed 2 years ago

AdrianLC commented 5 years ago

Hi, we are having a rust panic crash with one of our binary files. It happens with symbolic 6.1.3 and 6.1.4 and we are calling this from python with:

archive = symbolic.Archive.open(filename)
list(archive.iter_objects())  # it crashes on the third next() call on this iterable

Sharing the panic message below. I do not know any Rust so if you are missing any other info please say so and I will try to help.

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/home/appuser/.local/lib/python3.7/site-packages/symbolic/debuginfo.py", line 40, in iter_objects
    yield self._get_object(idx)
  File "/home/appuser/.local/lib/python3.7/site-packages/symbolic/debuginfo.py", line 61, in _get_object
    ptr = self._methodcall(lib.symbolic_archive_get_object, idx)
  File "/home/appuser/.local/lib/python3.7/site-packages/symbolic/utils.py", line 58, in _methodcall
    return rustcall(func, self._get_objptr(), *args)
  File "/home/appuser/.local/lib/python3.7/site-packages/symbolic/utils.py", line 93, in rustcall
    raise exc
symbolic.Panic: symbolic panicked: thread 'unnamed' panicked with 'index 9660037 out of range for slice of length 7340032' at src/libcore/slice/mod.rs:2413

stacktrace: stack backtrace:
   0: failure::backtrace::Backtrace::new::h8ada5e7dec1f1df0 (0x7f40621ea527)
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/backtrace/internal.rs:44
      <failure::backtrace::Backtrace as core::default::Default>::default::h52408e978655c67d
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/backtrace/mod.rs:125
   1: <T as core::convert::Into<U>>::into::h4de2394ec093dae3 (0x7f4062239404)
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/error/error_impl.rs:19
      symbolic::utils::set_panic_hook::{{closure}}::h45f5cafd1f9528b0
             at src/utils.rs:83
   2: rust_panic_with_hook (0x7f40621f40a4)
             at src/libstd/panicking.rs:482
   3: continue_panic_fmt (0x7f40621f3dde)
             at src/libstd/panicking.rs:385
   4: rust_begin_unwind (0x7f40621ff795)
   5: panic_fmt (0x7f406220375b)
             at src/libcore/panicking.rs:85
   6: slice_index_len_fail (0x7f40622031e4)
             at src/libcore/slice/mod.rs:2413
   7: symbolic_debuginfo::macho::MachArchive::object_by_index::h3a50fa549059d24f (0x7f4062100cf7)
             at /rustc/6c2484dc3c532c052f159264e970278d8b77cdc9/src/libcore/slice/mod.rs:2590
      symbolic_debuginfo::object::Archive::object_by_index::h3df0ae4733c276a0
             at /work/debuginfo/src/object.rs:504
   8: symbolic_archive_get_object (0x7f40622287b4)
             at src/debuginfo.rs:64
   9: ffi_call_unix64 (0x7f40625efdef)
  10: ffi_call (0x7f40625ef857)
             at ../src/x86/ffi64.c:525
  11: cdata_call (0x7f4062812663)
             at c/_cffi_backend.c:3086
  12: PyObject_Call (0x7f406a11d8c3)
             at Objects/call.c:245
  13: do_call_core (0x7f406a18ec2b)
             at Python/ceval.c:4645
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3191
  14: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  15: _PyFunction_FastCallDict (0x7f406a11d173)
             at Objects/call.c:376
  16: do_call_core (0x7f406a18ec2b)
             at Python/ceval.c:4645
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3191
  17: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  18: _PyFunction_FastCallKeywords (0x7f406a11ca11)
             at Objects/call.c:433
  19: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  20: function_code_fastcall (0x7f406a11c969)
             at Objects/call.c:283
      _PyFunction_FastCallKeywords
             at Objects/call.c:408
  21: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  22: gen_send_ex (0x7f406a122699)
             at Objects/genobject.c:221
      gen_iternext
             at Objects/genobject.c:542
  23: builtin_next (0x7f406a18b788)
             at Python/bltinmodule.c:1426
  24: _PyMethodDef_RawFastCallKeywords (0x7f406a11b9e2)
             at Objects/call.c:655
  25: _PyCFunction_FastCallKeywords (0x7f406a11b84f)
             at Objects/call.c:734
  26: call_function (0x7f406a192e65)
             at Python/ceval.c:4568
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3124
  27: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  28: PyEval_EvalCodeEx (0x7f406a18b9b8)
             at Python/ceval.c:3959
  29: PyEval_EvalCode (0x7f406a18b97a)
             at Python/ceval.c:524
  30: builtin_exec_impl (0x7f406a205743)
             at Python/bltinmodule.c:1079
      builtin_exec
             at Python/clinic/bltinmodule.c.h:283
  31: _PyMethodDef_RawFastCallKeywords (0x7f406a11b9e2)
             at Objects/call.c:655
  32: _PyCFunction_FastCallKeywords (0x7f406a11b84f)
             at Objects/call.c:734
  33: call_function (0x7f406a192e65)
             at Python/ceval.c:4568
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3124
  34: function_code_fastcall (0x7f406a11c969)
             at Objects/call.c:283
      _PyFunction_FastCallKeywords
             at Objects/call.c:408
  35: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  36: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  37: _PyFunction_FastCallKeywords (0x7f406a11ca11)
             at Objects/call.c:433
  38: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  39: function_code_fastcall (0x7f406a11c969)
             at Objects/call.c:283
      _PyFunction_FastCallKeywords
             at Objects/call.c:408
  40: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  41: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  42: _PyFunction_FastCallKeywords (0x7f406a11ca11)
             at Objects/call.c:433
  43: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  44: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  45: _PyFunction_FastCallKeywords (0x7f406a11ca11)
             at Objects/call.c:433
  46: call_function (0x7f406a18d918)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3139
  47: function_code_fastcall (0x7f406a11c969)
             at Objects/call.c:283
      _PyFunction_FastCallKeywords
             at Objects/call.c:408
  48: call_function (0x7f406a18d03a)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3124
  49: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  50: _PyFunction_FastCallDict (0x7f406a11d395)
             at Objects/call.c:376
  51: _PyObject_Call_Prepend (0x7f406a11d6a5)
             at Objects/call.c:908
  52: PyObject_Call (0x7f406a11d8c3)
             at Objects/call.c:245
  53: do_call_core (0x7f406a18ec2b)
             at Python/ceval.c:4645
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3191
  54: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  55: _PyFunction_FastCallDict (0x7f406a11d395)
             at Objects/call.c:376
  56: _PyObject_Call_Prepend (0x7f406a11d6a5)
             at Objects/call.c:908
  57: PyObject_Call (0x7f406a11d8c3)
             at Objects/call.c:245
  58: do_call_core (0x7f406a18ec2b)
             at Python/ceval.c:4645
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3191
  59: function_code_fastcall (0x7f406a11c969)
             at Objects/call.c:283
      _PyFunction_FastCallKeywords
             at Objects/call.c:408
  60: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  61: function_code_fastcall (0x7f406a11c969)
             at Objects/call.c:283
      _PyFunction_FastCallKeywords
             at Objects/call.c:408
  62: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  63: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  64: _PyFunction_FastCallKeywords (0x7f406a11ca11)
             at Objects/call.c:433
  65: call_function (0x7f406a18d03a)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3124
  66: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  67: PyEval_EvalCodeEx (0x7f406a18b9b8)
             at Python/ceval.c:3959
  68: PyEval_EvalCode (0x7f406a18b97a)
             at Python/ceval.c:524
  69: builtin_exec_impl (0x7f406a205743)
             at Python/bltinmodule.c:1079
      builtin_exec
             at Python/clinic/bltinmodule.c.h:283
  70: _PyMethodDef_RawFastCallKeywords (0x7f406a11b9e2)
             at Objects/call.c:655
  71: _PyCFunction_FastCallKeywords (0x7f406a11b84f)
             at Objects/call.c:734
  72: call_function (0x7f406a192e65)
             at Python/ceval.c:4568
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3124
  73: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  74: PyEval_EvalCodeEx (0x7f406a18b9b8)
             at Python/ceval.c:3959
  75: PyEval_EvalCode (0x7f406a18b97a)
             at Python/ceval.c:524
  76: builtin_exec_impl (0x7f406a205743)
             at Python/bltinmodule.c:1079
      builtin_exec
             at Python/clinic/bltinmodule.c.h:283
  77: _PyMethodDef_RawFastCallKeywords (0x7f406a11b9e2)
             at Objects/call.c:655
  78: _PyCFunction_FastCallKeywords (0x7f406a11b84f)
             at Objects/call.c:734
  79: call_function (0x7f406a192e65)
             at Python/ceval.c:4568
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3124
  80: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  81: _PyFunction_FastCallKeywords (0x7f406a11ca11)
             at Objects/call.c:433
  82: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  83: function_code_fastcall (0x7f406a11c969)
             at Objects/call.c:283
      _PyFunction_FastCallKeywords
             at Objects/call.c:408
  84: call_function (0x7f406a18ce21)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3110
  85: function_code_fastcall (0x7f406a11c969)
             at Objects/call.c:283
      _PyFunction_FastCallKeywords
             at Objects/call.c:408
  86: call_function (0x7f406a190861)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3093
  87: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  88: PyEval_EvalCodeEx (0x7f406a18b9b8)
             at Python/ceval.c:3959
  89: PyEval_EvalCode (0x7f406a18b97a)
             at Python/ceval.c:524
  90: builtin_exec_impl (0x7f406a205743)
             at Python/bltinmodule.c:1079
      builtin_exec
             at Python/clinic/bltinmodule.c.h:283
  91: _PyMethodDef_RawFastCallKeywords (0x7f406a11b9e2)
             at Objects/call.c:655
  92: _PyCFunction_FastCallKeywords (0x7f406a11b84f)
             at Objects/call.c:734
  93: call_function (0x7f406a190703)
             at Python/ceval.c:4568
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3124
  94: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  95: _PyFunction_FastCallKeywords (0x7f406a11ca11)
             at Objects/call.c:433
  96: call_function (0x7f406a18d03a)
             at Python/ceval.c:4616
      _PyEval_EvalFrameDefault
             at Python/ceval.c:3124
  97: _PyEval_EvalCodeWithName (0x7f406a18bcb0)
             at Python/ceval.c:3930
  98: _PyFunction_FastCallDict (0x7f406a11d1be)
             at Objects/call.c:376
  99: pymain_run_module (0x7f406a216a50)
             at Modules/main.c:320
 100: pymain_run_python (0x7f406a214975)
             at Modules/main.c:2864
      pymain_main
             at Modules/main.c:3028
 101: _Py_UnixMain (0x7f406a21460d)
             at Modules/main.c:3063
 102: __libc_start_main (0x7f4069c7e09a)
 103: _start (0x56544b0d6089)
 104: <unknown> (0x0)

Also is there any way to recover from the panic? I tried except symbolic.Panic but we still lose the parent process with a core dump.

AdrianLC commented 5 years ago

Also is there any way to recover from the panic? I tried except symbolic.Panic but we still lose the parent process with a core dump

We noticed the wheel on pip is built with python2.7 and we managed to fix this by building everything ourselves for python3.7 with rust 1.34. So at least it doesn't crash with a core dump now

jan-auer commented 5 years ago

@AdrianLC would it be possible to share the dSYM with me (via email to jan.auer@sentry.io, for instance)? I'm curious what might cause this panic -- especially since there shouldn't be a large difference between our binary wheels and a custom build.

Also, I'm curious as to why there is a core dump. The library catches the panic down, which is why you can see it along with the Python stack trace.

Would love to take a detailled look at a repro case or the affected file.

AdrianLC commented 5 years ago

Thank you @jan-auer. I just sent you the dSYM by email.

We generally cannot see the panic when it crashes. But for some reason I managed to see it when running the code from a shell. Also, it sort of seems to be fixed with our build and we get an exception instead. (Sort of, because we suspect it still might be happening with other zips)

AdrianLC commented 4 years ago

Hi, with the 7.1.0 release our regression test has changed from symbolic.Panic to this 

symbolic.ObjectErrorBadMachOobject: failed to process macho file
  caused by: invalid MachO file
  caused by: type is too big (3074949) for 754944

stacktrace: stack backtrace:
   0: failure::backtrace::Backtrace::new
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/backtrace/internal.rs:44
      <failure::backtrace::Backtrace as core::default::Default>::default
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/backtrace/mod.rs:125
   1: symbolic_archive_get_object
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.5/src/error/error_impl.rs:19

I guess the change comes from https://github.com/getsentry/symbolic/pull/169/files maybe it's enough to close this issue.

Swatinem commented 2 years ago

Looking at the original stack trace, I have the assumption this was an wrapping arithmetic problem that I found and fixed some time ago both coming from another customer issue, as well as using fuzzing.

The second stack trace reported later on looks different.

Since this issue is already over 2 years old, I will close this for now, and ask you to reopen if this is still a problem, preferably with a failing testcase that we can look at.