Random node crashes with Pyroscope enabled

[sandreim]

The pyroscope feature is not enabled by default or for release builds.

2022-03-08 15:50:39.600  INFO tokio-runtime-worker substrate: 💤 Idle (29 peers), best: #480596 (0xfc82…dcd3), finalized #480593 (0x3ad0…eb7b), ⬇ 43.3kiB/s ⬆ 60.7kiB/s    

====================

Version: 0.9.16-7314d62-x86_64-linux-gnu

   0: sp_panic_handler::panic_hook
             at usr/local/cargo/git/checkouts/substrate-7e08433d4c370a21/529c2fd/primitives/panic-handler/src/lib.rs:166:18
      sp_panic_handler::set::{{closure}}
             at usr/local/cargo/git/checkouts/substrate-7e08433d4c370a21/529c2fd/primitives/panic-handler/src/lib.rs:62:12
   1: std::panicking::rust_panic_with_hook
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:628:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:519:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:139:18
   4: rust_begin_unwind
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
   5: core::panicking::panic_fmt
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
   6: core::panicking::panic
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:50:5
   7: pyroscope::backends::pprof::fold
             at usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/pyroscope-0.3.0/src/backends/pprof.rs:108:26
      <pyroscope::backends::pprof::Pprof as pyroscope::backends::Backend>::report
             at usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/pyroscope-0.3.0/src/backends/pprof.rs:86:9
   8: pyroscope::pyroscope::PyroscopeAgent::_start::{{closure}}
             at usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/pyroscope-0.3.0/src/pyroscope.rs:301:30
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:123:18
   9: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/mod.rs:483:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
      std::panicking::try
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
      std::panic::catch_unwind
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
      std::thread::Builder::spawn_unchecked::{{closure}}
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/mod.rs:482:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:227:5
  10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/alloc/src/boxed.rs:1691:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/alloc/src/boxed.rs:1691:9
      std::sys::unix::thread::Thread::new::thread_start
             at rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys/unix/thread.rs:106:17
  11: start_thread
  12: clone

Thread '<unnamed>' panicked at 'attempt to subtract with overflow', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/pyroscope-0.3.0/src/backends/pprof.rs:108

[drahnr]

Pyroscope 0.3.1 addresses this, I thought I updated it to 0.3.1 already.

EDIT: https://github.com/paritytech/polkadot/pull/4871 does contain such a commit

[sandreim]

Hmm, this happening for a profiler node which runs with https://github.com/paritytech/polkadot/pull/4871: 4871-0.9.16-7314d62-f91cc88f

[drahnr]

But the backtrace clearly shows the outdated pyroscope version, something looks fishy.

Pyroscope was only merged as 25adb1627756b8fe1bb557b6a4ac85a0938a73e1 which is after 0.9.16 got released.

[sandreim]

Yep, the commit hash looks different, so likely this is before the 0.3.1 bump. I'll upgrade to check.

[sandreim]

The issue is resolved by #4871. Closing this.