search

google / stenographer

Stenographer is a packet capture solution which aims to quickly spool all packets to disk, then provide simple, fast access to subsets of those packets. Discussion/announcements at stenographer@googlegroups.com
Apache License 2.0
1.79k stars 238 forks source link

Feature Request: Utility to dump offline stenographer sensor data to pcap #204

Open charlesfair opened 5 years ago

charlesfair commented 5 years ago

It would be quite helpful to have a simple command-line utility that would dump a stenographer repository to pcap. This would really help with training and prototyping scenarios. This is a common used case with RockNSM. It is used in a training scenario, where it is used to monitor for a limited time, then later on all saved stenographer data needs to be shared out for further training.

aeppert commented 5 years ago

@charlesfair please look at the following script an see if it, at least in the interim, fits your needs:

https://github.com/aeppert/shell_scripts/blob/master/dump_steno.sh

charlesfair commented 5 years ago

Thank you @aeppert for the script! That will be much easier than doing it by hand.

One issue with using stenoread for the extraction is how when it is highly optimized to extract small, concise PCAP exactions. When I use it to export a large amount of PCAP, it basically takes over all the cores on the system where it is running. For instance on my test system, which was a quad-core I7 that was hyperthreading with 16GB RAM, stenoread was using around 700% of the CPU percentage, on all cores. I tried to do a full 24 hour PCAP export so as to not fill up the hard drive, but the stenoread process would effectively keep the Stenographer sensor, and Bro, from reliability capturing data during the export.

Basically it is a really really bad idea to export a large amount of Stenographer data as PCAP on a system that is actively collecting network data.

@dcode was interested if the author of Stenographer could create a utility to simply export massive amounts of PCAP without hammering the system during the export

Reference: issue https://github.com/google/stenographer/issues/162 @gconnell "In general, Steno is optimized to store lots of packets and return a small, targeted set. Note that if you're returning any appreciable percentage of the packets you store, Steno is probably thrashing your disk VERY hard."

The "small utility" that @dcode is interested in would be optimized specifically to simply export a large set of all packets, between two date/time ranges or such that would not thrash your disk.

MaxDiOrio commented 4 years ago

This is exactly what I'm interested in now. With no ability to trim captures in steno, or to cut off elephant flows, data consumption is huge. I'd need 170TB of storage space to store 30 days worth of captures in our environment.

If we can take the oldest capture files and convert them into pcaps so we can run trimpcap on them and then store them off somewhere else, that would be huge.

charlesfair commented 4 years ago

The feature your describing would be similar to Time Machine, and a strongly welcome addition.

https://github.com/zeek/time-machine

Charles "Chuck" A. Fair

On Jul 24, 2020, at 6:16 PM, Max DiOrio notifications@github.com wrote:

 This is exactly what I'm interested in now. With no ability to trim captures in steno, or to cut off elephant flows, data consumption is huge. I'd need 170TB of storage space to store 30 days worth of captures in our environment.

If we can take the oldest capture files and convert them into pcaps so we can run trimpcap on them and then store them off somewhere else, that would be huge.