gcla / termshark

A terminal UI for tshark, inspired by Wireshark
MIT License
9.02k stars 394 forks source link

Unable to Quit (from long-running sessions) #162

Open iam-TJ opened 11 months ago

iam-TJ commented 11 months ago

Problem

This may be related to to issue #121 which was thought to be fixed.

Debian 12 Bookworm, amd64, termshark 2.4.0

Current Behavior

I've recently been leaving termshark running for many hours and on 2 separate hosts had situations where despite going through the UI Quit workflow, and either :q!, Quit > Keep, or Quit > Delete, and the UI simply continuing as if the command hasn't been issued. The log seems to indicate that the background processes are being killed though:

time="2023-10-16T21:41:39+01:00" level=info msg="Started pcap command /usr/bin/tshark tshark -r /home/tj/.cache/termshark/pcaps/usb3hubKY688--2023-10-16--17-21-56.pcap -x
-Y '(frame.number >= 866001) and (frame.number < 867002)' with pid 1249168"
time="2023-10-16T21:41:39+01:00" level=info msg="Started PDML command /usr/bin/tshark tshark -T pdml -r /home/tj/.cache/termshark/pcaps/usb3hubKY688--2023-10-16--17-21-56.
pcap --color -Y '(frame.number >= 866001) and (frame.number < 867002)' with pid 1249169"
time="2023-10-17T14:04:55+01:00" level=info msg="Sending SIGKILL to 1090037: tail -f -c +0 /home/tj/.cache/termshark/pcaps/usb3hubKY688--2023-10-16--17-21-56.pcap"
time="2023-10-17T14:04:55+01:00" level=info msg="Sending SIGKILL to 1090026: termshark -i usb3hubKY688 -w /home/tj/.cache/termshark/pcaps/usb3hubKY688--2023-10-16--17-21-5
6.pcap -f 'ip or ip6 or arp'"
time="2023-10-17T14:04:55+01:00" level=info msg="Sending SIGKILL to 1090027: tshark -T psml -o gui.column.format:\\\"No.\\\",\\\"%m\\\",\\\"No.\\\",\\\"%m\\\",\\\"Time\\\"
,\\\"%t\\\",\\\"Source\\\",\\\"%s\\\",\\\"Dest\\\",\\\"%d\\\",\\\"Proto\\\",\\\"%p\\\",\\\"Length\\\",\\\"%L\\\",\\\"Info\\\",\\\"%i\\\" -r - -l --color"
time="2023-10-17T14:04:55+01:00" level=info msg="Load operation complete"
time="2023-10-17T14:06:51+01:00" level=error msg="Could not delete file /home/tj/.cache/termshark/pcaps/usb3hubKY688--2023-10-16--17-21-56.pcap: remove /home/tj/.cache/ter
mshark/pcaps/usb3hubKY688--2023-10-16--17-21-56.pcap: no such file or directory"
time="2023-10-17T14:07:34+01:00" level=info msg="Requested stop psml + iface"
time="2023-10-17T14:07:35+01:00" level=info msg="Requested stop psml + iface"
time="2023-10-17T14:07:38+01:00" level=info msg="Requested stop psml + iface"
time="2023-10-17T14:15:22+01:00" level=warning msg="Could not read wireshark config folder /home/tj/.config/wireshark/profiles: open /home/tj/.config/wireshark/profiles: no such file or directory"
time="2023-10-17T14:16:12+01:00" level=warning msg="Could not read wireshark config folder /home/tj/.config/wireshark/profiles: open /home/tj/.config/wireshark/profiles: no such file or directory"
time="2023-10-17T14:16:15+01:00" level=warning msg="Could not read wireshark config folder /home/tj/.config/wireshark/profiles: open /home/tj/.config/wireshark/profiles: no such file or directory"
time="2023-10-17T14:18:35+01:00" level=info msg="Requested stop psml + iface"

This session had to be KILLed.

Expected Behavior

For short-duration sessions quit does work - I don't have a handle on what period is required to trigger this as yet but safe to say it is likely 6 hours+.

Steps to Reproduce

Leave running for long periods.

Context

Please provide the complete output of these commands:

Copyright 1998-2023 Gerald Combs gerald@wireshark.org and contributors. Licensed under the terms of the GNU General Public License (version 2 or later). This is free software; see the file named COPYING in the distribution. There is NO WARRANTY; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiled (64-bit) using GCC 12.2.0, with GLib 2.74.6, with PCRE2, with zlib 1.2.13, with libpcap, with POSIX capabilities (Linux), with libnl 3, with Lua 5.2.4, with GnuTLS 3.7.9 and PKCS #11 support, with Gcrypt 1.10.1, with Kerberos (MIT), with MaxMind, with nghttp2 1.52.0, with brotli, with LZ4, with Zstandard, with Snappy, with libxml2 2.9.14, with libsmi 0.4.8, with binary plugins.

Running on Linux 6.5.1+debian+tj, with AMD Ryzen 7 3700X 8-Core Processor (with SSE4.2), with 31997 MB of physical memory, with GLib 2.74.6, with PCRE2 10.42 2022-12-11, with zlib 1.2.13, with libpcap 1.10.3 (with TPACKET_V3), with c-ares 1.18.1, with GnuTLS 3.7.9, with Gcrypt 1.10.1, with nghttp2 1.52.0, with brotli 1.0.9, with LZ4 1.9.4, with Zstandard 1.5.4, with libsmi 0.4.8, with LC_TYPE=en_GB.UTF-8, binary plugins supported.



I've attached `gdb` to it but as yet not got anything useful; if you could suggest some locations for breakpoints I can then single-step from there when it is likely to fail. I've tried that with a well-behaved short-duration session starting from `ui/ui.go:479` so am reasonably familiar with a 'good' quit flow.
bretello commented 8 months ago

I'm seeing the same issue. Here is a full log with debug = true obtained using

curl "http://localhost:6060/debug/pprof/goroutine?debug=1"
pprof/goroutine?debug=1" ```console $ curl "http://localhost:6060/debug/pprof/goroutine?debug=1" goroutine profile: total 21 3 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a2227b51 0x55c2a2228305 0x55c2a2341bf6 0x55c2a234c625 0x55c2a221a641 # 0x55c2a2227b50 io.(*pipe).read+0xb0 io/pipe.go:57 # 0x55c2a2228304 io.(*PipeReader).Read+0x24 io/pipe.go:136 # 0x55c2a2341bf5 bufio.(*Scanner).Scan+0x875 bufio/scan.go:214 # 0x55c2a234c624 github.com/sirupsen/logrus.(*Entry).writerScanner+0xa4 github.com/sirupsen/logrus@v1.7.0/writer.go:59 2 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a277237a 0x55c2a26a55a2 0x55c2a221a641 # 0x55c2a2772379 github.com/gcla/termshark/v2/widgets/filter.New.func7+0x159 github.com/gcla/termshark/v2/widgets/filter/filter.go:330 # 0x55c2a26a55a1 github.com/gcla/termshark/v2.TrackedGo.func1+0xc1 github.com/gcla/termshark/v2/utils.go:559 2 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a277257d 0x55c2a26a55a2 0x55c2a221a641 # 0x55c2a277257c github.com/gcla/termshark/v2/widgets/filter.New.func6+0xfc github.com/gcla/termshark/v2/widgets/filter/filter.go:300 # 0x55c2a26a55a1 github.com/gcla/termshark/v2.TrackedGo.func1+0xc1 github.com/gcla/termshark/v2/utils.go:559 1 @ 0x55c2a21ba5b4 0x55c2a2216a6f 0x55c2a23044f9 0x55c2a221a641 # 0x55c2a2216a6e os/signal.signal_recv+0x2e runtime/sigqueue.go:152 # 0x55c2a23044f8 os/signal.loop+0x18 os/signal/signal_unix.go:23 1 @ 0x55c2a21ddf76 0x55c2a22144e5 0x55c2a2361c15 0x55c2a2361a2d 0x55c2a235e9ab 0x55c2a28a6d45 0x55c2a28a78de 0x55c2a25e238f 0x55c2a25e3d29 0x55c2a25e578c 0x55c2a25e0e67 0x55c2a221a641 # 0x55c2a22144e4 runtime/pprof.runtime_goroutineProfileWithLabels+0x24 runtime/mprof.go:846 # 0x55c2a2361c14 runtime/pprof.writeRuntimeProfile+0xb4 runtime/pprof/pprof.go:723 # 0x55c2a2361a2c runtime/pprof.writeGoroutine+0x4c runtime/pprof/pprof.go:683 # 0x55c2a235e9aa runtime/pprof.(*Profile).WriteTo+0x14a runtime/pprof/pprof.go:330 # 0x55c2a28a6d44 net/http/pprof.handler.ServeHTTP+0x4a4 net/http/pprof/pprof.go:253 # 0x55c2a28a78dd net/http/pprof.Index+0x13d net/http/pprof/pprof.go:371 # 0x55c2a25e238e net/http.HandlerFunc.ServeHTTP+0x2e net/http/server.go:2109 # 0x55c2a25e3d28 net/http.(*ServeMux).ServeHTTP+0x148 net/http/server.go:2487 # 0x55c2a25e578b net/http.serverHandler.ServeHTTP+0x30b net/http/server.go:2947 # 0x55c2a25e0e66 net/http.(*conn).serve+0x606 net/http/server.go:1991 1 @ 0x55c2a21e8a76 0x55c2a21b501b 0x55c2a21b4b18 0x55c2a22b7bca 0x55c2a26b7289 0x55c2a26c862f 0x55c2a26a55a2 0x55c2a221a641 # 0x55c2a22b7bc9 os/exec.(*Cmd).Wait+0x1c9 os/exec/exec.go:608 # 0x55c2a26b7288 github.com/gcla/termshark/v2/pkg/pcap.(*Command).Wait+0x28 github.com/gcla/termshark/v2/pkg/pcap/cmds.go:62 # 0x55c2a26c862e github.com/gcla/termshark/v2/pkg/pcap.(*InterfaceLoader).loadIfacesSync.func2+0x2e github.com/gcla/termshark/v2/pkg/pcap/loader.go:2099 # 0x55c2a26a55a1 github.com/gcla/termshark/v2.TrackedGo.func1+0xc1 github.com/gcla/termshark/v2/utils.go:559 1 @ 0x55c2a21e8a76 0x55c2a21e1537 0x55c2a22149a9 0x55c2a2283f32 0x55c2a228529a 0x55c2a2285288 0x55c2a2290a1e 0x55c2a2290a16 0x55c2a233a005 0x55c2a2338d6e 0x55c2a221a641 # 0x55c2a22149a8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:305 # 0x55c2a2283f31 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84 # 0x55c2a2285299 internal/poll.(*pollDesc).waitRead+0x259 internal/poll/fd_poll_runtime.go:89 # 0x55c2a2285287 internal/poll.(*FD).Read+0x247 internal/poll/fd_unix.go:167 # 0x55c2a2290a1d os.(*File).read+0x5d os/file_posix.go:31 # 0x55c2a2290a15 os.(*File).Read+0x55 os/file.go:118 # 0x55c2a233a004 github.com/gdamore/tcell/v2.(*devTty).Read+0x24 github.com/gdamore/tcell/v2@v2.5.0/tty_unix.go:47 # 0x55c2a2338d6d github.com/gdamore/tcell/v2.(*tScreen).inputLoop+0xcd github.com/gdamore/tcell/v2@v2.5.0/tscreen.go:1633 1 @ 0x55c2a21e8a76 0x55c2a21e1537 0x55c2a22149a9 0x55c2a2283f32 0x55c2a22891b4 0x55c2a22891a1 0x55c2a23e76b5 0x55c2a23fd4c8 0x55c2a23fc69d 0x55c2a25e5ce5 0x55c2a25e591d 0x55c2a28a8ec5 0x55c2a28a8e95 0x55c2a221a641 # 0x55c2a22149a8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:305 # 0x55c2a2283f31 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84 # 0x55c2a22891b3 internal/poll.(*pollDesc).waitRead+0x233 internal/poll/fd_poll_runtime.go:89 # 0x55c2a22891a0 internal/poll.(*FD).Accept+0x220 internal/poll/fd_unix.go:614 # 0x55c2a23e76b4 net.(*netFD).accept+0x34 net/fd_unix.go:172 # 0x55c2a23fd4c7 net.(*TCPListener).accept+0x27 net/tcpsock_posix.go:142 # 0x55c2a23fc69c net.(*TCPListener).Accept+0x3c net/tcpsock.go:288 # 0x55c2a25e5ce4 net/http.(*Server).Serve+0x384 net/http/server.go:3070 # 0x55c2a25e591c net/http.(*Server).ListenAndServe+0x7c net/http/server.go:2999 # 0x55c2a28a8ec4 net/http.ListenAndServe+0x44 net/http/server.go:3255 # 0x55c2a28a8e94 main.cmain.func2+0x14 github.com/gcla/termshark/v2/cmd/termshark/termshark.go:558 1 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a2227f52 0x55c2a2228425 0x55c2a2227586 0x55c2a2226544 0x55c2a22b6bda 0x55c2a22b6bc6 0x55c2a22b7985 0x55c2a221a641 # 0x55c2a2227f51 io.(*pipe).write+0x211 io/pipe.go:86 # 0x55c2a2228424 io.(*PipeWriter).Write+0x24 io/pipe.go:165 # 0x55c2a2227585 io.(*multiWriter).Write+0x85 io/multi.go:85 # 0x55c2a2226543 io.copyBuffer+0x203 io/io.go:429 # 0x55c2a22b6bd9 io.Copy+0x39 io/io.go:386 # 0x55c2a22b6bc5 os/exec.(*Cmd).writerDescriptor.func1+0x25 os/exec/exec.go:407 # 0x55c2a22b7984 os/exec.(*Cmd).Start.func1+0x24 os/exec/exec.go:544 1 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a2336aaa 0x55c2a23739b9 0x55c2a221a641 # 0x55c2a2336aa9 github.com/gdamore/tcell/v2.(*tScreen).PollEvent+0x69 github.com/gdamore/tcell/v2@v2.5.0/tscreen.go:1052 # 0x55c2a23739b8 github.com/gcla/gowid.(*App).StartTCellEvents.func1+0x78 github.com/gcla/gowid@v1.4.0/app.go:580 1 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a23387cc 0x55c2a221a641 # 0x55c2a23387cb github.com/gdamore/tcell/v2.(*tScreen).mainLoop+0x16b github.com/gdamore/tcell/v2@v2.5.0/tscreen.go:1572 1 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a233a59f 0x55c2a221a641 # 0x55c2a233a59e github.com/gdamore/tcell/v2.(*devTty).Start.func1+0xbe github.com/gdamore/tcell/v2@v2.5.0/tty_unix.go:92 1 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a26c7fdb 0x55c2a26a55a2 0x55c2a221a641 # 0x55c2a26c7fda github.com/gcla/termshark/v2/pkg/pcap.(*InterfaceLoader).loadIfacesSync.func3+0x21a github.com/gcla/termshark/v2/pkg/pcap/loader.go:2140 # 0x55c2a26a55a1 github.com/gcla/termshark/v2.TrackedGo.func1+0xc1 github.com/gcla/termshark/v2/utils.go:559 1 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a26e601c 0x55c2a26a55a2 0x55c2a221a641 # 0x55c2a26e601b github.com/gcla/termshark/v2/pkg/confwatcher.New.func1+0x11b github.com/gcla/termshark/v2/pkg/confwatcher/confwatcher.go:49 # 0x55c2a26a55a1 github.com/gcla/termshark/v2.TrackedGo.func1+0xc1 github.com/gcla/termshark/v2/utils.go:559 1 @ 0x55c2a21e8a76 0x55c2a21f8bbc 0x55c2a28add5a 0x55c2a28a8b3a 0x55c2a21e86b3 0x55c2a221a641 # 0x55c2a28add59 main.cmain+0x4bd9 github.com/gcla/termshark/v2/cmd/termshark/termshark.go:1156 # 0x55c2a28a8b39 main.main+0xf9 github.com/gcla/termshark/v2/cmd/termshark/termshark.go:70 # 0x55c2a21e86b2 runtime.main+0x212 runtime/proc.go:250 1 @ 0x55c2a226bb67 0x55c2a2269ac5 0x55c2a22852c5 0x55c2a22852ad 0x55c2a2285065 0x55c2a23e55a9 0x55c2a23f4765 0x55c2a25dabff 0x55c2a221a641 # 0x55c2a226bb66 syscall.Syscall+0x26 syscall/syscall_linux.go:68 # 0x55c2a2269ac4 syscall.read+0x44 syscall/zsyscall_linux_amd64.go:696 # 0x55c2a22852c4 syscall.Read+0x284 syscall/syscall_unix.go:183 # 0x55c2a22852ac internal/poll.ignoringEINTRIO+0x26c internal/poll/fd_unix.go:794 # 0x55c2a2285064 internal/poll.(*FD).Read+0x24 internal/poll/fd_unix.go:163 # 0x55c2a23e55a8 net.(*netFD).Read+0x28 net/fd_posix.go:55 # 0x55c2a23f4764 net.(*conn).Read+0x44 net/net.go:183 # 0x55c2a25dabfe net/http.(*connReader).backgroundRead+0x3e net/http/server.go:678 1 @ 0x55c2a226bbf6 0x55c2a2303658 0x55c2a26b331d 0x55c2a26b25f5 0x55c2a221a641 # 0x55c2a226bbf5 syscall.Syscall6+0x35 syscall/syscall_linux.go:90 # 0x55c2a2303657 golang.org/x/sys/unix.EpollWait+0x57 golang.org/x/sys@v0.0.0-20220520151302-bc2c85ada10a/unix/zsyscall_linux_amd64.go:56 # 0x55c2a26b331c gopkg.in/fsnotify/fsnotify%2ev1.(*fdPoller).wait+0x7c gopkg.in/fsnotify/fsnotify.v1@v1.4.7/inotify_poller.go:86 # 0x55c2a26b25f4 gopkg.in/fsnotify/fsnotify%2ev1.(*Watcher).readEvents+0x274 gopkg.in/fsnotify/fsnotify.v1@v1.4.7/inotify.go:192 ```

Something else that I've noticed is that sometimes I am able to quit and the UI goes away, but the shell becomes completely unresponsive (possibly due to termshark intercepting ctrl-c and ctrl-z signals). Not sure if it's related, but I cannot seem to reproduce it right now.