search

cleesmith / unifiedbeat

Unifiedbeat reads records from Unified2 binary files generated by network intrusion detection software and indexes the records in Elasticsearch.
MIT License
30 stars 14 forks source link

Not sure if unifiedbeat is working or not. #8

Closed dajoen closed 8 years ago

dajoen commented 8 years ago

I,m testing unifiedbeat and am whondering if it is working with my snort setup.

The beat quits with the following log:

2016/06/06 15:17:56.783770 u2beat.go:123: INFO Setup: Rules warnings: 0 multiple line rules rejected, 9441 duplicate rules rejected 2016/06/06 15:17:56.783793 u2beat.go:124: INFO Setup: Rules stats: 122 rule files read, 8756 rules created 2016/06/06 15:17:56.787930 u2beat.go:140: INFO Setup: registrar: registry file: "/usr/lib/golang/src/github.com/cleesmith/unifiedbeat/.unifiedbeat" 2016/06/06 15:17:56.787952 u2beat.go:141: INFO Setup: registrar: file source: "" 2016/06/06 15:17:56.787964 u2beat.go:142: INFO Setup: registrar: file offset: 0 2016/06/06 15:17:56.788288 beat.go:267: INFO unifiedbeat sucessfully setup. Start running. 2016/06/06 15:17:56.788315 u2beat.go:148: INFO Run: start spooling and publishing... 2016/06/06 15:17:56.788328 u2spoolandpublish.go:52: INFO U2SpoolAndPublish: spooling and publishing... 2016/06/06 15:17:56.802475 u2spoolandpublish.go:96: CRIT U2SpoolAndPublish: unexpected error: 'unexpected EOF' 2016/06/06 15:17:56.802751 u2beat.go:182: INFO Run: updated registry file. 2016/06/06 15:17:56.802769 beat.go:307: INFO Start exiting beat 2016/06/06 15:17:56.807845 beat.go:282: INFO Stopping Beat 2016/06/06 15:17:56.807873 u2beat.go:192: INFO Stop: is spooling and publishing running? 'false' 2016/06/06 15:17:56.807892 u2beat.go:213: INFO Stop: done after waiting 16.801µs. 2016/06/06 15:17:56.807904 beat.go:290: INFO Cleaning up unifiedbeat before shutting down. 2016/06/06 15:17:56.807916 u2beat.go:217: INFO Cleanup: is spooling and publishing running? 'false' 2016/06/06 15:17:56.807965 u2beat.go:221: INFO Cleanup: closed GeoIp2Reader. 2016/06/06 15:17:56.807977 u2beat.go:223: INFO Cleanup: done.

This is my config:

sensor:
  # note: unified2 files must be timestamped with
  # a file extension similar to ".1452978988",
  # this is the typical set up for Snort/Suricata

  # where are the unified2 files for this sensor:
  unified2_path: "/var/log/snort/"

  # what is the unified2 filename prefix, which may be anything,
  # but there is no globbing, so do not use "*" on the end:
  unified2_prefix: "snort.log"

  # use GeoLite2 or GeoIP2 database for both IPv4/6 addresses:
  geoip2_path: "/var/GeoIP/GeoLite2-City.mmdb"

  # where are the Rules (signatures):
  rules:
    # gen_msg_map must be a single file reference, no glob's
    gen_msg_map_path: "/etc/snort/gen-msg.map"

    # rules path may be a glob
    # make sure no file is defined twice as this can
    # lead to lots of duplicate rule warnings:
    paths:
      - "/etc/snort/rules/*.rules"

  # add fixed/known details about this sensor:
  fields:
    sensor_hostname: QAH-VM-MON01
    sensor_interface: eth0
    sensor_type: snort

  # Set to true to store the additional fields (above) as top level fields instead
  # of under the "fields" sub-dictionary. In case of name conflicts with the
  # fields added by unifiedbeat itself, the custom fields overwrite the default fields.
  fields_under_root: true

  # Configure spool timeout to wait for spool/publish to gracefully terminate.
  # The default is 5 seconds, increase if spool/publish takes longer to finish.
  #spooler_timeout: 1

I would expect the beat to keep running and tail the current snort log file.

cleesmith commented 8 years ago

It does read and tail valid unified2 format binary files from both Snort and Suricata. I see this error message in your log messages:

2016/06/06 15:17:56.802475 u2spoolandpublish.go:96: CRIT U2SpoolAndPublish: unexpected error: 'unexpected EOF'

Usually this means the file named snort.log.??? is not a valid unified2 or is corrupt. Have you verified the file using U2SpewFoo, and that your /etc/snort/snort.conf file contains:

output unified2: filename snort.log, limit 128, mpls_event_types, vlan_event_types
dajoen commented 8 years ago

Thanks, I had those defined. But after further investigation I've had to disable some options in the default CentOS configuration in order for the output in unified2 to work. Changed those, and it is working now. Although I'm running into a new problem.

2016/06/08 13:53:08.398994 u2beat.go:123: INFO Setup: Rules warnings: 0 multiple line rules rejected, 9441 duplicate rules rejected 2016/06/08 13:53:08.399019 u2beat.go:124: INFO Setup: Rules stats: 122 rule files read, 8756 rules created 2016/06/08 13:53:08.400264 u2beat.go:140: INFO Setup: registrar: registry file: "/usr/lib/golang/src/github.com/cleesmith/unifiedbeat/.unifiedbeat" 2016/06/08 13:53:08.400290 u2beat.go:141: INFO Setup: registrar: file source: "/var/log/snort/snort.log" 2016/06/08 13:53:08.400303 u2beat.go:142: INFO Setup: registrar: file offset: 122965 2016/06/08 13:53:08.400566 beat.go:267: INFO unifiedbeat sucessfully setup. Start running. 2016/06/08 13:53:08.400585 u2beat.go:148: INFO Run: start spooling and publishing... 2016/06/08 13:53:08.400598 u2spoolandpublish.go:52: INFO U2SpoolAndPublish: spooling and publishing... fatal error: runtime: out of memory

runtime stack: runtime.throw(0xb595c0, 0x16) /usr/local/go/src/runtime/panic.go:530 +0x90 runtime.sysMap(0xc822800000, 0x1cac0000, 0x0, 0x13bf518) /usr/local/go/src/runtime/mem_linux.go:206 +0x9b runtime.(_mheap).sysAlloc(0xe90ec0, 0x1cac0000, 0x175360e88) /usr/local/go/src/runtime/malloc.go:429 +0x191 runtime.(_mheap).grow(0xe90ec0, 0xe560, 0x0) /usr/local/go/src/runtime/mheap.go:651 +0x63 runtime.(_mheap).allocSpanLocked(0xe90ec0, 0xe55f, 0x40de24) /usr/local/go/src/runtime/mheap.go:553 +0x4f6 runtime.(_mheap).alloc_m(0xe90ec0, 0xe55f, 0x100000000, 0x100e93150) /usr/local/go/src/runtime/mheap.go:437 +0x119 runtime.(_mheap).alloc.func1() /usr/local/go/src/runtime/mheap.go:502 +0x41 runtime.systemstack(0xc820123ee0) /usr/local/go/src/runtime/asm_amd64.s:307 +0xab runtime.(_mheap).alloc(0xe90ec0, 0xe55f, 0x10100000000, 0x7f1779427198) /usr/local/go/src/runtime/mheap.go:503 +0x63 runtime.largeAlloc(0x1cabcbbf, 0x1, 0x7f17752ca6e0) /usr/local/go/src/runtime/malloc.go:766 +0xb3 runtime.mallocgc.func3() /usr/local/go/src/runtime/malloc.go:664 +0x33 runtime.systemstack(0xc820019500) /usr/local/go/src/runtime/asm_amd64.s:291 +0x79 runtime.mstart() /usr/local/go/src/runtime/proc.go:1048

I'll if I can tweak that, and open a new ticket if needed.