retis-org / retis

Tracing packets in the Linux networking stack & friends
https://retis.readthedocs.io/en/stable/
102 stars 14 forks source link

ovs collector panicked while creating a bridge #73

Closed vlrpl closed 1 year ago

vlrpl commented 1 year ago

a panic was observed while creating an ovs bridge with the tool already running. Below the trace:

RUST_BACKTRACE=full ./target/debug/packet-tracer collect -c ovs
18:01:52 [INFO] Attaching probe to usdt /usr/local/sbin/ovs-vswitchd:dpif_netlink_operate__:op_flow_execute
thread '<unnamed>' panicked at 'attempt to subtract with overflow', src/core/user/proc.rs:384:62
stack backtrace:
   0:     0x55a2cf25d2c0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb280c2b0faedb192
   1:     0x55a2cf27a93e - core::fmt::write::h30e0b7ef777337ad
   2:     0x55a2cf25ac35 - std::io::Write::write_fmt::h86627e30c2b512b3
   3:     0x55a2cf25d085 - std::sys_common::backtrace::print::h7ed0882ed869c236
   4:     0x55a2cf25e90f - std::panicking::default_hook::{{closure}}::h9a127e13324a150a
   5:     0x55a2cf25e64a - std::panicking::default_hook::hf8f07fa1688cedd2
   6:     0x55a2cf25f008 - std::panicking::rust_panic_with_hook::he6d410a49c1deab2
   7:     0x55a2cf25ed61 - std::panicking::begin_panic_handler::{{closure}}::h3a4af972edd4df52
   8:     0x55a2cf25d76c - std::sys_common::backtrace::__rust_end_short_backtrace::h04151587e1857959
   9:     0x55a2cf25eac2 - rust_begin_unwind
  10:     0x55a2cef548d3 - core::panicking::panic_fmt::h5085b5d784b56c67
  11:     0x55a2cef549ad - core::panicking::panic::h699f7acfe9b26bc1
  12:     0x55a2cef89d1c - packet_tracer::core::user::proc::Process::get_note_from_symbol::h42392f1df81038e3
                               at /home/pvalerio/workspace/open_source/github/net-trace/packet-tracer-vlrpl/src/core/user/proc.rs:384:62
  13:     0x55a2cef99fef - packet_tracer::core::probe::user::user::register_unmarshaler::{{closure}}::h7174ef40370fcc82
                               at /home/pvalerio/workspace/open_source/github/net-trace/packet-tracer-vlrpl/src/core/probe/user/user.rs:98:24
  14:     0x55a2cefc75f5 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h0eaa2820cd54c2e2
                               at /builddir/build/BUILD/rustc-1.66.1-src/library/alloc/src/boxed.rs:2001:9
  15:     0x55a2cefe6120 - packet_tracer::core::events::bpf::parse_raw_event::h299ad2fecfa5aede
                               at /home/pvalerio/workspace/open_source/github/net-trace/packet-tracer-vlrpl/src/core/events/bpf.rs:262:25
  16:     0x55a2ceffa733 - packet_tracer::core::events::bpf::BpfEvents::start_polling::{{closure}}::h94dc8d997d7c95c3
                               at /home/pvalerio/workspace/open_source/github/net-trace/packet-tracer-vlrpl/src/core/events/bpf.rs:132:31
  17:     0x55a2cf018e34 - libbpf_rs::ringbuf::RingBufferBuilder::call_sample_cb::h6d1234f729d02c26
                               at /home/pvalerio/.cargo/git/checkouts/libbpf-rs-a64433d6203387de/52ab250/libbpf-rs/src/ringbuf.rs:128:9
  18:     0x55a2cf04cd31 - ringbuf_process_ring
                               at /home/pvalerio/.cargo/registry/src/github.com-1ecc6299db9ec823/libbpf-sys-1.0.4+v1.0.1/libbpf/src/ringbuf.c:231:11
  19:     0x55a2cf04ce31 - ring_buffer__poll
                               at /home/pvalerio/.cargo/registry/src/github.com-1ecc6299db9ec823/libbpf-sys-1.0.4+v1.0.1/libbpf/src/ringbuf.c:288:9
  20:     0x55a2cf018ee4 - libbpf_rs::ringbuf::RingBuffer::poll::h211593462a5b2144
                               at /home/pvalerio/.cargo/git/checkouts/libbpf-rs-a64433d6203387de/52ab250/libbpf-rs/src/ringbuf.rs:157:28
  21:     0x55a2ceffacfc - packet_tracer::core::events::bpf::BpfEvents::start_polling::{{closure}}::h016bc39abb859420
                               at /home/pvalerio/workspace/open_source/github/net-trace/packet-tracer-vlrpl/src/core/events/bpf.rs:158:17
  22:     0x55a2cefa51a1 - std::sys_common::backtrace::__rust_begin_short_backtrace::hafb370250f6afa6b
                               at /builddir/build/BUILD/rustc-1.66.1-src/library/std/src/sys_common/backtrace.rs:121:18
  23:     0x55a2cef78e01 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc858349d835e0303
                               at /builddir/build/BUILD/rustc-1.66.1-src/library/std/src/thread/mod.rs:551:17
  24:     0x55a2cefe3e51 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hdb80aa3a4e01895b
                               at /builddir/build/BUILD/rustc-1.66.1-src/library/core/src/panic/unwind_safe.rs:271:9
  25:     0x55a2cef8bd11 - std::panicking::try::do_call::hfc928c58770113b0
                               at /builddir/build/BUILD/rustc-1.66.1-src/library/std/src/panicking.rs:483:40
  26:     0x55a2cef8bebb - __rust_try
  27:     0x55a2cef8ba5f - std::panicking::try::hc2c9b75d3499bfe0
                               at /builddir/build/BUILD/rustc-1.66.1-src/library/std/src/panicking.rs:447:19
  28:     0x55a2cef7c401 - std::panic::catch_unwind::h72fc8bbca879c25b
                               at /builddir/build/BUILD/rustc-1.66.1-src/library/std/src/panic.rs:137:14
  29:     0x55a2cef7872c - std::thread::Builder::spawn_unchecked_::{{closure}}::h6537250acbf18c1c
                               at /builddir/build/BUILD/rustc-1.66.1-src/library/std/src/thread/mod.rs:550:30
  30:     0x55a2cefea8ee - core::ops::function::FnOnce::call_once{{vtable.shim}}::h09a1881c17915317
                               at /builddir/build/BUILD/rustc-1.66.1-src/library/core/src/ops/function.rs:251:5
  31:     0x55a2cf261b53 - std::sys::unix::thread::Thread::new::thread_start::hfad602368217ab7c
  32:     0x7f3686b8e12d - start_thread
  33:     0x7f3686c0fbc0 - clone3
amorenoz commented 1 year ago

@vlrpl are you able to reproduce it easily? I cannot on my system

vlrpl commented 1 year ago

@vlrpl are you able to reproduce it easily? I cannot on my system

just tried again with the current ovs master.

> uname -r
6.1.9-200.fc37.x86_64

> ovs-vsctl --version
ovs-vsctl (Open vSwitch) 3.1.90
DB Schema 8.3.1
> ldd `which ovs-vswitchd`
        linux-vdso.so.1 (0x00007ffeee15a000)
        libssl.so.3 => /lib64/libssl.so.3 (0x00007f6d7ce54000)
        libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f6d7ca00000)
        libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007f6d7ce4a000)
        libatomic.so.1 => /lib64/libatomic.so.1 (0x00007f6d7ce40000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f6d7c920000)
        libunbound.so.8 => /lib64/libunbound.so.8 (0x00007f6d7c7c7000)
        libunwind.so.8 => /lib64/libunwind.so.8 (0x00007f6d7c7ae000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f6d7c5d1000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f6d7c5b7000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6d7cf1c000)
        libprotobuf-c.so.1 => /lib64/libprotobuf-c.so.1 (0x00007f6d7ce32000)
        libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007f6d7c55e000)
        libpython3.11.so.1.0 => /lib64/libpython3.11.so.1.0 (0x00007f6d7be00000)
        libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f6d7c533000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6d7c513000)

it only happens if you create the first bridge. Relaunching the tool and creating one more bridge doesn't trigger the issue.