search

google / stenographer

Stenographer is a packet capture solution which aims to quickly spool all packets to disk, then provide simple, fast access to subsets of those packets. Discussion/announcements at stenographer@googlegroups.com
Apache License 2.0
1.79k stars 238 forks source link

[Ubuntu] Stenotype keeps crashing #205

Closed FabFaeb closed 4 years ago

FabFaeb commented 5 years ago

Hi,

I am currently playing around with Stenographer on several Ubuntu 16.04/18.04 VMs. Unfortunately on all of these machines, I see recurring crashes of the stenotype component after 2 to 5 minutes (usually around the 2-minute-mark). The error looks like this (from /var/log/syslog):

Apr 2 07:51:48 nsm-sensor stenographer[699]: 2019-04-02T05:51:48.716656Z T:537fe7 [util.cc:117] WATCHDOG FAILURE: Thread 0 012ABORTABORTABORT#012/usr/bin/stenotype(+0x70a1) [0x5604943e40a1] 012/usr/bin/stenotype(+0x5240) [0x5604943e2240] 012/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd57f) [0x7effda83b57f] 012/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7effdad536db] 012/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7effda29688f] Apr 2 07:51:48 nsm-sensor stenographer[699]: 2019/04/02 07:51:48 Stenotype stopped after 2m52.433619838s: stenotype wait failed: signal: aborted (core dumped) Apr 2 07:51:48 nsm-sensor stenographer[699]: 2019/04/02 07:51:48 Deleted stale output file "/etc/stenographer/packets/.1554184258882421" Apr 2 07:56:43 nsm-sensor stenographer[699]: 2019-04-02T05:56:43.235697Z T:8cdec7 [util.cc:117] WATCHDOG FAILURE: Thread 0 012ABORTABORTABORT#012/usr/bin/stenotype(+0x70a1) [0x55bdabf000a1] 012/usr/bin/stenotype(+0x5240) [0x55bdabefe240] 012/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd57f) [0x7f8b0fa5d57f] 012/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f8b0ff756db] 012/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f8b0f4b888f] Apr 2 07:56:43 nsm-sensor stenographer[699]: 2019/04/02 07:56:43 Stenotype stopped after 4m54.482629796s: stenotype wait failed: signal: aborted (core dumped) Apr 2 07:56:43 nsm-sensor stenographer[699]: 2019/04/02 07:56:43 Deleted stale output file "/etc/stenographer/packets/.1554184554061973"

Additionally, the written files in the packets directory are all empty. Essentially I am having the same issues as described in this reddit post: https://www.reddit.com/r/AskNetsec/comments/9181kc/anyone_run_stenographer/

Has anyone else experienced similar issues? Are there any ideas what I could do about it? What I have already tried:

I'll be thankful for any hints!

lmangani commented 5 years ago

If nothing gets written to disk this might have to do with file permissions. Try running stenotype manually mimicking the options you see the service using, plus the -v FLAG to get more insight as of the crash

dhoelzer commented 5 years ago

Stenotype simply stops processing immediately, never throwing any errors or logging any packets:

2019-04-08T13:23:55.240558Z T:9479a7 [stenotype.cc:545] Stenotype running with these arguments:
2019-04-08T13:23:55.240610Z T:9479a7 [stenotype.cc:547] 0:  "stenotype"
2019-04-08T13:23:55.240615Z T:9479a7 [stenotype.cc:547] 1:  "--dir=/data/stenographer/packets"
2019-04-08T13:23:55.240620Z T:9479a7 [stenotype.cc:547] 2:  "--iface=enp2s0f1"
2019-04-08T13:23:55.240624Z T:9479a7 [stenotype.cc:547] 3:  "-vvvvvvvvvvv"
2019-04-08T13:23:55.240628Z T:9479a7 [stenotype.cc:549] Starting, page size is 4096
2019-04-08T13:23:55.240634Z T:9479a7 [stenotype.cc:576] Setting up AF_PACKET sockets for packet reading
2019-04-08T13:23:55.596129Z T:9479a7 [packets.cc:229] Setting promiscuous mode for enp2s0f1
2019-04-08T13:23:55.596169Z T:9479a7 [packets.cc:237] Interface enp2s0f1 already in promisc mode
2019-04-08T13:23:55.596234Z T:9479a7 [stenotype.cc:248] Dropping privileges
2019-04-08T13:23:55.596246Z T:9479a7 [stenotype.cc:253] Dropping priviledges from 0 to GID nogroup
2019-04-08T13:23:55.596503Z T:9479a7 [stenotype.cc:265] Dropping priviledges from 0 to UID nobody
2019-04-08T13:23:55.598026Z T:9479a7 [stenotype.cc:652] Starting writing threads
2019-04-08T13:23:55.598050Z T:9479a7 [stenotype.cc:655] Starting thread 0
2019-04-08T13:23:55.598074Z T:1473d7 [stenotype.cc:427] Handling signals
2019-04-08T13:23:55.598105Z T:9479a7 [stenotype.cc:666] Starting indexing threads
2019-04-08T13:23:55.598200Z T:1373b7 [stenotype.cc:395] Starting WriteIndexes thread 0
2019-04-08T13:23:55.598858Z T:9479a7 [stenotype.cc:680] ===============Waiting for thread==============
2019-04-08T13:23:55.598859Z T:1373b7 [stenotype.cc:403] Waiting for index
2019-04-08T13:23:55.598859Z T:13f3c7 [stenotype.cc:450] Thread 0 starting to process packets
2019-04-08T13:23:55.599117Z T:9479a7 [stenotype.cc:683] Thread finished
2019-04-08T13:23:55.599144Z T:9479a7 [stenotype.cc:686] Finished all threads
2019-04-08T13:23:55.599169Z T:9479a7 [stenotype.cc:689] Closing write index queue 0, waiting for thread
2019-04-08T13:23:55.599245Z T:1373b7 [stenotype.cc:405] Got index 0
2019-04-08T13:23:55.599259Z T:1373b7 [stenotype.cc:414] Exiting write index thread
2019-04-08T13:23:56.598772Z T:9479a7 [stenotype.cc:693] Index thread finished
2019-04-08T13:23:56.598803Z T:9479a7 [stenotype.cc:698] Process exiting successfully
2019-04-08T13:23:56.598952Z T:1473d7 [stenotype.cc:436] Signal handling done
lmangani commented 5 years ago

This won't solve your issue but I would try running it manually to make sure data gets stored at all, ie: 

stenotype --preallocate_file_mb=20 --blocks=128 --seccomp=none -v --threads=1 --iface=enp2s0f1 --dir=/data/stenographer --fileage_sec=1200 --blockage_sec=30
dhoelzer commented 5 years ago

This won't solve your issue but I would try running it manually to make sure data gets stored at all, ie:

stenotype --preallocate_file_mb=20 --blocks=128 --seccomp=none -v --threads=1 --iface=enp2s0f1 --dir=/data/stenographer --fileage_sec=1200 --blockage_sec=30

That doesn't fix it but better reveals the problem. Seeing that it is failing to create directories got me to look at the ID that it's dropping to. Contrary to the documentation, it is dropping to nobody and nogroup rather than stenographer, which is what I would expect since the INSTALL.MD directs you to chown -R stenographer:stenographer the path for the repository.

FabFaeb commented 5 years ago

Sorry for my delayed answer: In my case it seemed to be a combination of a file permission problem and seccomp. When deactivating seccomp and setting the appropriate permissions, stenographer works as intended. When I keep seccomp activated I get the following lines in the syslog and stenotype just dies. 

kernel: [  452.433303] audit: type=1326 audit(1554283284.929:18): auid=4294967295 uid=111 gid=117 ses=4294967295 pid=23371 comm="stenotype" exe="/usr/bin/stenotype" sig=31 arch=c000003e syscall=2 compat=0 ip=0x7fc10ab3aa5f code=0x0

kernel: [  453.434763] audit: type=1326 audit(1554283285.933:19): auid=4294967295 uid=111 gid=117 ses=4294967295 pid=23369 comm="stenotype" exe="/usr/bin/stenotype" sig=31 arch=c000003e syscall=231 compat=0 ip=0x7fc10ab8d748 code=0x0

Is my understanding, that these two syscalls (231 and 2) need to be added to stenotype.cc via SECCOMP_RULE_ADD, correct?

Shaligram commented 4 years ago

I am getting this error while trying to run. any help much appreciated. This is with default options also. Do we need to run from specific user. I am running from root:root

stenotype --preallocate_file_mb=20 --blocks=128 --seccomp=none -v --threads=1 --iface=ens224 --dir=/data/stenographer --fileage_sec=1200 --blockage_sec=30 2020-06-24T06:07:42.421984Z T:653ba7 [stenotype.cc:554] Starting, page size is 4096 2020-06-24T06:07:42.422128Z T:653ba7 [stenotype.cc:581] Setting up AF_PACKET sockets for packet reading 2020-06-24T06:07:42.447663Z T:653ba7 [stenotype.cc:252] Dropping privileges 2020-06-24T06:07:42.447695Z T:653ba7 [stenotype.cc:257] Dropping priviledges from 0 to GID nogroup 2020-06-24T06:07:42.448809Z T:653ba7 [stenotype.cc:260] CHECK(group != NULL) Unable to get info for group nogroup ABORTABORTABORT stenotype() [0x4081f8] stenotype() [0x42bb14] stenotype() [0x42e1bc] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fc86373d545] stenotype() [0x40412b]

Aborted (core dumped)

FabFaeb commented 4 years ago

For me, stenographer seems to be working as expected when I run it via systemd after setting the right file permissions for the data directories. I can even enable seccomp after adding the two new rules I introduced in PR #225 . I think your problems might be different from mine and are probably worth their own issue. For this reason, I will close this issue now.

zwatts00 commented 2 years ago

@FabFaeb Would you be able to clarify what file permissions for which directories you changed? I believe I am having this issue on Ubuntu 20.04 with stenographer installed from apt (0.0~git20190917.0.3545c54-1build1)

I have tried to verify the permissions match the ones modified in install.sh but to no avail.

Here are the errors I am getting, maybe it is a different issue:

Jun 16 20:31:55 l3c-ids stenographer[66445]: 2022/06/16 20:31:55 Stenotype stopped after 2m0.755949623s: stenotype wait failed: signal: aborted (core dumped)
Jun 16 20:31:55 l3c-ids stenographer[66445]: /usr/sbin/stenotype(+0x7ebb) [0x55cedae42ebb]
                                             /usr/sbin/stenotype(+0x7956) [0x55cedae42956]
                                             /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f9b19898de4]
                                             /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f9b199ce609]
                                             /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f9b196d2133]
Jun 16 20:31:55 l3c-ids stenographer[66445]: 2022-06-16T20:31:55.316355Z T:97c517 [util.cc:117] WATCHDOG FAILURE: Thread 0
                                             ABORTABORTABORT
FabFaeb commented 2 years ago

@zwatts00 Since this was quite some time ago and I am longer working with stenographer, I am not able to say what exactly I changed - sorry. However, I am relatively sure that I changed permissions on the "data-directory", where the pcaps should be written. I would suggest to (temporarily) chmod 777 this directory and then run with --seccomp=none to see if this is related to this issue.