dougburks
commented
7 years ago Hi @petiepooo ,
This may already be fixed in the new version of CapMe for the Elastic stack. It just searches 5 minutes before and 5 minutes after, so it should be much more precise in its targeting: https://github.com/dougburks/elastic-test/blob/master/var/www/so/capme/.inc/callback-elk.php#L207-L210
petiepooo
I am very happy that UDP is working in capme now, but DNS queries are still causing me issues. On a fairly busy site, where DNS query source ports are reused a few times per hour, pivoting to capme from a BRO_CONN or BRO_INTEL entry in ELSA (of a DNS query+response) pulls up query/response pairs unrelated to the connection I am querying. For example, given the following BRO_INTEL line:
Mon May 29 22:21:51 1496110910.029995|CtIUPiMHnDMA89vl4|10.X.X.X|58894|208.67.222.222|53|sinkhole.cert.pl|Intel::DOMAIN|DNS::IN_REQUEST|myhost-mon1-4|Intel::DOMAIN|mysrc|-|-|- host=127.0.0.1 program=bro_intel class=BRO_INTEL srcip=10.X.X.X srcport=58894 dstip=208.67.222.222 dstport=53 indicator=sinkhole.cert.pl indicator_type=Intel::DOMAIN seen_where=DNS::IN_REQUEST sources=mysrc
If I pivot to capme, it fills in the query fields with the appropriate src/dst IP and port, and a timestamp range of 1496109111 to 1496112711. That includes 30 minutes before and after the event time of 1496110910.029995 as given in ELSA. When that query runs, capme returns with three hits in this particular instance, with timestamps of 1496109444.970557, 1496109451.802349, and 1496109593.786958, all in the first few seconds of the time range and well before the timestamp of the query I'm interested in.
If I then close the capme output page, change the start time to 1496110910 as given in the event, and resubmit, it returns the same entries, being that its using archived data as stored in the /nsm/server_data/securityonion/archive/2017-05-30/ directory. This is because that archive only separates by ip/port/proto and date, with no consideration of the start time.
If I close the capme output page AND manually remove the raw archive file, then resubmit, only then do I get the connection I'm actually interested in. In this case, it's 1496110910.030033 (the connection of interest) and 1496111020.204885 (another one a little while later using the same src port).
FWIW, the dailylog files around those times on my system are: -rw-r--r-- 1 sguil sguil 165414965 May 30 01:56 snort.log.1496109085 -rw-r--r-- 1 sguil sguil 173388421 May 30 02:00 snort.log.1496109367 -rw-r--r-- 1 sguil sguil 182712576 May 30 02:02 snort.log.1496109632 -rw-r--r-- 1 sguil sguil 169390451 May 30 02:07 snort.log.1496109779 -rw-r--r-- 1 sguil sguil 165715407 May 30 02:12 snort.log.1496110071 -rw-r--r-- 1 sguil sguil 166100549 May 30 02:18 snort.log.1496110365 -rw-r--r-- 1 sguil sguil 163215086 May 30 02:20 snort.log.1496110685 -rw-r--r-- 1 sguil sguil 168493158 May 30 02:25 snort.log.1496110843 -rw-r--r-- 1 sguil sguil 168791782 May 30 02:30 snort.log.1496111121 -rw-r--r-- 1 sguil sguil 166299025 May 30 02:35 snort.log.1496111444 -rw-r--r-- 1 sguil sguil 165297167 May 30 02:41 snort.log.1496111749 -rw-r--r-- 1 sguil sguil 165032449 May 30 02:45 snort.log.1496112089 -rw-r--r-- 1 sguil sguil 165140580 May 30 02:48 snort.log.1496112356 -rw-r--r-- 1 sguil sguil 165412680 May 30 02:50 snort.log.1496112493 -rw-r--r-- 1 sguil sguil 165417181 May 30 02:56 snort.log.1496112652 -rw-r--r-- 1 sguil sguil 166509389 May 30 03:01 snort.log.1496112972
This is using securityonion-capme version 20121213-0ubuntu0securityonion65.
I'm not a TCL developer, so am not very comfortable digging into the code, but it seems like it's searching the first pcap in the time range (or maybe the first one that returns a hit?), and then just ignoring the rest of them.
Would it be possible to add a link on the output page giving the user the ability to clear the archive file? That would ease the workaround of resubmitting with a more precise timestamp.
Alternatively, would it be possible to add the epoch timestamps (or at least the start epoch) to the archive filename so that multiple src/dst/proto tuples can be cached per day?
Ultimately, though, I would think the fix would be to better define the start/end timestamps and/or allow user configuration of them, and search all files within the specified range.