search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.2k stars 3.5k forks source link

Can't find sincedb or successfully start Logstash as a service. #4075

Open joshusre opened 8 years ago

joshusre commented 8 years ago

Setup: I'm on a clean install of CentOS7 minimal, and I've installed logstash 1.5 and elasticsearch1.44 via yum

Problem: Logstash won't index data from /var/log/mylogs without it being started manually at command line. No errors are present in logstash.err, no data in logstash.log to help, and nothing in ES logs.

I thought this was interesting (saved from console of logstash on manual start):

{:timestamp=>"2015-10-21T08:57:56.694000-0400", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"/root/.sincedb_8f309eb34476af59efaabf28f6aac73a", :path=>["/var/log/pythonapps/.log", "/var/log/pythonapps/.log.*"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"120", :method=>"register"}

But, I can't find .sincedb, on my box, at all (tried various forms, checked all home folders, find, locate, ls -a | less, etc). I'm also told that because logstash may be starting as something other than user logstash, that might be the cause. But, /etc/init.d/logstash:

LS_USER=logstash LS_GROUP=logstash

And, that I doubt that would explain the console output?

I'm out of ideas why logstash can't init correctly and has to be started manually, instead of as a service "systemctl start elasticsearch.service"

Bug? My fault? Need more docs / data?

Thanks for your time.

(Please give me up to 24 hours to reply. This is for work, I don't have remote, and it's super frustrating trying to reply to a thread that's been closed prematurely.)

joshusre commented 8 years ago

Did I post this wrong or something?

jsvd commented 8 years ago

Can you post your configuration file? my guess would be that logstash is trying to start as use logstash but the files in /var/log/ aren't accessible to that use.

Also, looking at the log line you posted, the :path is strange: :path=>["/var/log/python_apps/.log", "/var/log/python_apps/.log.*"

Perhaps /var/log/python_apps/.log should be /var/log/python_apps/*.log and "/var/log/python_apps/.log.* should be /var/log/python_apps/*.log.*?

joshusre commented 8 years ago

Thanks for your reply. My configuration file, as of now, looks like this (the paths were for testing, now I'm relying on an rsyslog instance on another box sending everything to LogStash:

input { syslog { type => "remote_syslog" port => 5514 } }

output { elasticsearch { host => "192.168.250.106" protocol => "http" cluster => "elasticeagle" } stdout { codec => rubydebug } }

When I try to run logstash from bash "service logstash start" or similar, it doesn't do anything. But, if I launch it via bin/logstash -f config_file everything works as expected.

Is there any more information I may provide you to help diagnose this problem?

jsvd commented 8 years ago

can you show the logfile for logstash when running as a service? are you config files in the right place? can you change the init.d/logstash file to run with --debug?

joshusre commented 8 years ago

(I'm including my commands in case I've made a mistake)

Conf file is /etc/logstash/conf.d/logstash_es.conf (my tutorial implied that this could vary in name as long as it was .conf)

Logfile for logstash when running as a service (service logstash start) has nothing in it after about 3 minutes. Stdout has "Sending logstash logs to /var/log/logstash/logstash.log". Logstash.err is empty.

After adding --debug to LS_OPTS and restarting the logstash service with the config from above:

{:timestamp=>"2015-10-27T08:27:39.136000-0400", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn} {:timestamp=>"2015-10-27T08:27:49.969000-0400", :message=>"Reading config file", :file=>"logstash/agent.rb", :level=>:debug, :line=>"309", :method=>"local_config"} {:timestamp=>"2015-10-27T08:27:50.028000-0400", :message=>"Compiled pipeline code:\n @inputs = []\n @filters = []\n @outputs = []\n @periodic_flushers = []\n @shutdown_flushers = []\n\n @input_syslog_1 = plugin(\"input\", \"syslog\", LogStash::Util.hash_merge_many({ \"type\" => (\"remote_syslog\") }, { \"port\" => 5514 }))\n\n @inputs << @input_syslog_1\n\n @output_elasticsearch_2 = plugin(\"output\", \"elasticsearch\", LogStash::Util.hash_merge_many({ \"host\" => (\"192.168.250.111\") }, { \"protocol\" => (\"http\") }, { \"cluster\" => (\"elasticeagle\") }))\n\n @outputs << @output_elasticsearch_2\n\n @output_stdout_3 = plugin(\"output\", \"stdout\", LogStash::Util.hash_merge_many({ \"codec\" => (\"rubydebug\") }))\n\n @outputs << @output_stdout_3\n\n def filter_func(event)\n events = [event]\n @logger.debug? && @logger.debug(\"filter received\", :event => event.to_hash)\n events\n end\n def output_func(event)\n @logger.debug? && @logger.debug(\"output received\", :event => event.to_hash)\n @output_elasticsearch_2.handle(event)\n @output_stdout_3.handle(event)\n \n end", :level=>:debug, :file=>"logstash/pipeline.rb", :line=>"29", :method=>"initialize"} {:timestamp=>"2015-10-27T08:27:50.030000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"syslog", :path=>"logstash/inputs/syslog", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"} {:timestamp=>"2015-10-27T08:27:50.726000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"} {:timestamp=>"2015-10-27T08:27:50.740000-0400", :message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.765000-0400", :message=>"config LogStash::Inputs::Syslog/@type = \"remote_syslog\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.766000-0400", :message=>"config LogStash::Inputs::Syslog/@port = 5514", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.767000-0400", :message=>"config LogStash::Inputs::Syslog/@debug = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.768000-0400", :message=>"config LogStash::Inputs::Syslog/@codec = <LogStash::Codecs::Plain charset=>\"UTF-8\">", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.769000-0400", :message=>"config LogStash::Inputs::Syslog/@add_field = {}", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.770000-0400", :message=>"config LogStash::Inputs::Syslog/@host = \"0.0.0.0\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.771000-0400", :message=>"config LogStash::Inputs::Syslog/@use_labels = true", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.772000-0400", :message=>"config LogStash::Inputs::Syslog/@facility_labels = [\"kernel\", \"user-level\", \"mail\", \"system\", \"security/authorization\", \"syslogd\", \"line printer\", \"network news\", \"UUCP\", \"clock\", \"security/authorization\", \"FTP\", \"NTP\", \"log audit\", \"log alert\", \"clock\", \"local0\", \"local1\", \"local2\", \"local3\", \"local4\", \"local5\", \"local6\", \"local7\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.773000-0400", :message=>"config LogStash::Inputs::Syslog/@severity_labels = [\"Emergency\", \"Alert\", \"Critical\", \"Error\", \"Warning\", \"Notice\", \"Informational\", \"Debug\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.774000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"output", :name=>"elasticsearch", :path=>"logstash/outputs/elasticsearch", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"} {:timestamp=>"2015-10-27T08:27:50.996000-0400", :message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.998000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@host = [\"192.168.250.111\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.998000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@protocol = \"http\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:50.999000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@cluster = \"elasticeagle\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.000000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@type = \"\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.018000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.019000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@exclude_tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.020000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@codec = <LogStash::Codecs::Plain charset=>\"UTF-8\">", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.020000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@workers = 1", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.021000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@index = \"logstash-%{+YYYY.MM.dd}\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.021000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@manage_template = true", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.022000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@template_name = \"logstash\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.023000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@template_overwrite = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.023000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@embedded = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.024000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@embedded_http_port = \"9200-9300\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.025000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@max_inflight_requests = 50", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.025000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@flush_size = 5000", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.026000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@idle_flush_time = 1", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.027000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@action = \"index\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.027000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@path = \"/\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.028000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@ssl = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.043000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@ssl_certificate_verification = true", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.044000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@sniffing = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.044000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@max_retries = 3", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.045000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@retry_max_items = 5000", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.046000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@retry_max_interval = 5", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.047000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@doc_as_upsert = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.055000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@upsert = \"\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.056000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"} {:timestamp=>"2015-10-27T08:27:51.109000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"rubydebug", :path=>"logstash/codecs/rubydebug", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"} {:timestamp=>"2015-10-27T08:27:51.119000-0400", :message=>"config LogStash::Codecs::RubyDebug/@metadata = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.120000-0400", :message=>"config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug metadata=>false>", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.120000-0400", :message=>"config LogStash::Outputs::Stdout/@type = \"\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.121000-0400", :message=>"config LogStash::Outputs::Stdout/@tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.122000-0400", :message=>"config LogStash::Outputs::Stdout/@exclude_tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.134000-0400", :message=>"config LogStash::Outputs::Stdout/@workers = 1", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.137000-0400", :message=>"config LogStash::Filters::Grok/@overwrite = [\"message\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.137000-0400", :message=>"config LogStash::Filters::Grok/@match = {\"message\"=>\"<%{POSINT:priority}>%{SYSLOGLINE}\"}", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.138000-0400", :message=>"config LogStash::Filters::Grok/@tag_on_failure = [\"_grokparsefailure_sysloginput\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.138000-0400", :message=>"config LogStash::Filters::Grok/@type = \"\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.139000-0400", :message=>"config LogStash::Filters::Grok/@tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.139000-0400", :message=>"config LogStash::Filters::Grok/@exclude_tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.140000-0400", :message=>"config LogStash::Filters::Grok/@add_tag = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.141000-0400", :message=>"config LogStash::Filters::Grok/@remove_tag = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.141000-0400", :message=>"config LogStash::Filters::Grok/@add_field = {}", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.142000-0400", :message=>"config LogStash::Filters::Grok/@remove_field = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.142000-0400", :message=>"config LogStash::Filters::Grok/@periodic_flush = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.143000-0400", :message=>"config LogStash::Filters::Grok/@patterns_dir = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.143000-0400", :message=>"config LogStash::Filters::Grok/@break_on_match = true", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.144000-0400", :message=>"config LogStash::Filters::Grok/@named_captures_only = true", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.160000-0400", :message=>"config LogStash::Filters::Grok/@keep_empty_captures = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.161000-0400", :message=>"config LogStash::Filters::Grok/@singles = true", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.202000-0400", :message=>"config LogStash::Filters::Date/@match = [\"timestamp\", \"MMM d HH:mm:ss\", \"MMM dd HH:mm:ss\", \"ISO8601\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.205000-0400", :message=>"config LogStash::Filters::Date/@locale = nil", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.205000-0400", :message=>"config LogStash::Filters::Date/@timezone = nil", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.206000-0400", :message=>"config LogStash::Filters::Date/@type = \"\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.207000-0400", :message=>"config LogStash::Filters::Date/@tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.207000-0400", :message=>"config LogStash::Filters::Date/@exclude_tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.214000-0400", :message=>"config LogStash::Filters::Date/@add_tag = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.215000-0400", :message=>"config LogStash::Filters::Date/@remove_tag = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.215000-0400", :message=>"config LogStash::Filters::Date/@add_field = {}", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.216000-0400", :message=>"config LogStash::Filters::Date/@remove_field = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.216000-0400", :message=>"config LogStash::Filters::Date/@periodic_flush = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.217000-0400", :message=>"config LogStash::Filters::Date/@target = \"@timestamp\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.218000-0400", :message=>"config LogStash::Filters::Date/@tag_on_failure = [\"_dateparsefailure\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"} {:timestamp=>"2015-10-27T08:27:51.253000-0400", :message=>"Grok patterns path", :patternsdir=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns", "/opt/logstash/patterns/"], :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"245", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.254000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/aws", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.256000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/bro", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.256000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/firewalls", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.257000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/grok-patterns", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.261000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/haproxy", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.261000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/java", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.262000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/junos", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.262000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/linux-syslog", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.263000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/mcollective", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.263000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/mcollective-patterns", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.264000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/mongodb", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.264000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/nagios", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.265000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/postgresql", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.265000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/rails", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.266000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/redis", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.266000-0400", :message=>"Grok loading patterns from file", :path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/ruby", :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"252", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.267000-0400", :message=>"Match data", :match=>{"message"=>"<%{POSINT:priority}>%{SYSLOGLINE}"}, :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"259", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.268000-0400", :message=>"Grok compile", :field=>"message", :patterns=>["<%{POSINT:priority}>%{SYSLOGLINE}"], :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"264", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.268000-0400", :message=>"regexp: /message", :pattern=>"<%{POSINT:priority}>%{SYSLOGLINE}", :level=>:debug, :file=>"logstash/filters/grok.rb", :line=>"266", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.280000-0400", :message=>"Adding pattern", "S3_REQUEST_LINE"=>"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.281000-0400", :message=>"Adding pattern", "S3_ACCESS_LOG"=>"%{WORD:owner} %{NOTSPACE:bucket} [%{HTTPDATE:timestamp}] %{IP:clientip} %{NOTSPACE:requester} %{NOTSPACE:request_id} %{NOTSPACE:operation} %{NOTSPACE:key} (?:\"%{S3_REQUEST_LINE}\"|-) (?:%{INT:response:int}|-) (?:-|%{NOTSPACE:error_code}) (?:%{INT:bytes:int}|-) (?:%{INT:object_size:int}|-) (?:%{INT:request_time_ms:int}|-) (?:%{INT:turnaround_time_ms:int}|-) (?:%{QS:referrer}|-) (?:\"?%{QS:agent}\"?|-) (?:-|%{NOTSPACE:version_id})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.282000-0400", :message=>"Adding pattern", "ELB_URIPATHPARAM"=>"%{URIPATH:path}(?:%{URIPARAM:params})?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.302000-0400", :message=>"Adding pattern", "ELBURI"=>"%{URIPROTO:proto}://(?:%{USER}(?::[^@])?@)?(?:%{URIHOST:urihost})?(?:%{ELB_URIPATHPARAM})?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.303000-0400", :message=>"Adding pattern", "ELB_REQUEST_LINE"=>"(?:%{WORD:verb} %{ELB_URI:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.303000-0400", :message=>"Adding pattern", "ELB_ACCESS_LOG"=>"%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:clientip}:%{INT:clientport:int} (?:(%{IP:backendip}:?:%{INT:backendport:int})|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:response:int} %{INT:backend_response:int} %{INT:received_bytes:int} %{INT:bytes:int} \"%{ELB_REQUEST_LINE}\"", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.306000-0400", :message=>"Adding pattern", "BRO_HTTP"=>"%{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{INT:trans_depth}\t%{GREEDYDATA:method}\t%{GREEDYDATA:domain}\t%{GREEDYDATA:uri}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:user_agent}\t%{NUMBER:request_body_len}\t%{NUMBER:response_body_len}\t%{GREEDYDATA:status_code}\t%{GREEDYDATA:status_msg}\t%{GREEDYDATA:info_code}\t%{GREEDYDATA:info_msg}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:bro_tags}\t%{GREEDYDATA:username}\t%{GREEDYDATA:password}\t%{GREEDYDATA:proxied}\t%{GREEDYDATA:orig_fuids}\t%{GREEDYDATA:orig_mime_types}\t%{GREEDYDATA:resp_fuids}\t%{GREEDYDATA:resp_mime_types}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.307000-0400", :message=>"Adding pattern", "BRO_DNS"=>"%{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{INT:trans_id}\t%{GREEDYDATA:query}\t%{GREEDYDATA:qclass}\t%{GREEDYDATA:qclass_name}\t%{GREEDYDATA:qtype}\t%{GREEDYDATA:qtype_name}\t%{GREEDYDATA:rcode}\t%{GREEDYDATA:rcode_name}\t%{GREEDYDATA:AA}\t%{GREEDYDATA:TC}\t%{GREEDYDATA:RD}\t%{GREEDYDATA:RA}\t%{GREEDYDATA:Z}\t%{GREEDYDATA:answers}\t%{GREEDYDATA:TTLs}\t%{GREEDYDATA:rejected}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.308000-0400", :message=>"Adding pattern", "BRO_CONN"=>"%{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{GREEDYDATA:service}\t%{NUMBER:duration}\t%{NUMBER:orig_bytes}\t%{NUMBER:resp_bytes}\t%{GREEDYDATA:conn_state}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:missed_bytes}\t%{GREEDYDATA:history}\t%{GREEDYDATA:orig_pkts}\t%{GREEDYDATA:orig_ip_bytes}\t%{GREEDYDATA:resp_pkts}\t%{GREEDYDATA:resp_ip_bytes}\t%{GREEDYDATA:tunnel_parents}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.308000-0400", :message=>"Adding pattern", "BRO_FILES"=>"%{NUMBER:ts}\t%{NOTSPACE:fuid}\t%{IP:tx_hosts}\t%{IP:rx_hosts}\t%{NOTSPACE:conn_uids}\t%{GREEDYDATA:source}\t%{GREEDYDATA:depth}\t%{GREEDYDATA:analyzers}\t%{GREEDYDATA:mime_type}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:duration}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:is_orig}\t%{GREEDYDATA:seen_bytes}\t%{GREEDYDATA:total_bytes}\t%{GREEDYDATA:missing_bytes}\t%{GREEDYDATA:overflow_bytes}\t%{GREEDYDATA:timedout}\t%{GREEDYDATA:parent_fuid}\t%{GREEDYDATA:md5}\t%{GREEDYDATA:sha1}\t%{GREEDYDATA:sha256}\t%{GREEDYDATA:extracted}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.313000-0400", :message=>"Adding pattern", "NETSCREENSESSIONLOG"=>"%{SYSLOGTIMESTAMP:date} %{IPORHOST:device} %{IPORHOST}: NetScreen device_id=%{WORD:device_id}%{DATA}: start_time=%{QUOTEDSTRING:start_time} duration=%{INT:duration} policy_id=%{INT:policy_id} service=%{DATA:service} proto=%{INT:proto} src zone=%{WORD:src_zone} dst zone=%{WORD:dst_zone} action=%{WORD:action} sent=%{INT:sent} rcvd=%{INT:rcvd} src=%{IPORHOST:src_ip} dst=%{IPORHOST:dst_ip} src_port=%{INT:src_port} dst_port=%{INT:dst_port} src-xlated ip=%{IPORHOST:src_xlated_ip} port=%{INT:src_xlated_port} dst-xlated ip=%{IPORHOST:dst_xlated_ip} port=%{INT:dst_xlated_port} session_id=%{INT:session_id} reason=%{GREEDYDATA:reason}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.313000-0400", :message=>"Adding pattern", "CISCO_TAGGED_SYSLOG"=>"^<%{POSINT:syslog_pri}>%{CISCOTIMESTAMP:timestamp}( %{SYSLOGHOST:sysloghost})?: %%{CISCOTAG:ciscotag}:", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.314000-0400", :message=>"Adding pattern", "CISCOTIMESTAMP"=>"%{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.314000-0400", :message=>"Adding pattern", "CISCOTAG"=>"[A-Z0-9]+-%{INT}-(?:[A-Z0-9]+)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.315000-0400", :message=>"Adding pattern", "CISCO_ACTION"=>"Built|Teardown|Deny|Denied|denied|requested|permitted|denied by ACL|discarded|est-allowed|Dropping|created|deleted", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.315000-0400", :message=>"Adding pattern", "CISCOREASON"=>"Duplicate TCP SYN|Failed to locate egress interface|Invalid transport field|No matching connection|DNS Response|DNS Query|(?:%{WORD}\s)_", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.316000-0400", :message=>"Adding pattern", "CISCO_DIRECTION"=>"Inbound|inbound|Outbound|outbound", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.317000-0400", :message=>"Adding pattern", "CISCO_INTERVAL"=>"first hit|%{INT}-second interval", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.317000-0400", :message=>"Adding pattern", "CISCO_XLATE_TYPE"=>"static|dynamic", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.318000-0400", :message=>"Adding pattern", "CISCOFW104001"=>"((?:Primary|Secondary)) Switching to ACTIVE - %{GREEDYDATA:switch_reason}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.318000-0400", :message=>"Adding pattern", "CISCOFW104002"=>"((?:Primary|Secondary)) Switching to STANDBY - %{GREEDYDATA:switch_reason}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.319000-0400", :message=>"Adding pattern", "CISCOFW104003"=>"((?:Primary|Secondary)) Switching to FAILED.", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.324000-0400", :message=>"Adding pattern", "CISCOFW104004"=>"((?:Primary|Secondary)) Switching to OK.", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.324000-0400", :message=>"Adding pattern", "CISCOFW105003"=>"((?:Primary|Secondary)) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} waiting", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.325000-0400", :message=>"Adding pattern", "CISCOFW105004"=>"((?:Primary|Secondary)) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} normal", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.326000-0400", :message=>"Adding pattern", "CISCOFW105005"=>"((?:Primary|Secondary)) Lost Failover communications with mate on [Ii]nterface %{GREEDYDATA:interface_name}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.326000-0400", :message=>"Adding pattern", "CISCOFW105008"=>"((?:Primary|Secondary)) Testing [Ii]nterface %{GREEDYDATA:interface_name}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.327000-0400", :message=>"Adding pattern", "CISCOFW105009"=>"((?:Primary|Secondary)) Testing on [Ii]nterface %{GREEDYDATA:interface_name} (?:Passed|Failed)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.327000-0400", :message=>"Adding pattern", "CISCOFW106001"=>"%{CISCO_DIRECTION:direction} %{WORD:protocol} connection %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{GREEDYDATA:tcp_flags} on interface %{GREEDYDATA:interface}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.328000-0400", :message=>"Adding pattern", "CISCOFW106006_106007_106010"=>"%{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} (?:from|src) %{IP:src_ip}/%{INT:src_port}((%{DATA:src_fwuser}))? (?:to|dst) %{IP:dst_ip}/%{INT:dst_port}((%{DATA:dst_fwuser}))? (?:on interface %{DATA:interface}|due to %{CISCO_REASON:reason})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.328000-0400", :message=>"Adding pattern", "CISCOFW106014"=>"%{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} src %{DATA:src_interface}:%{IP:src_ip}((%{DATA:src_fwuser}))? dst %{DATA:dst_interface}:%{IP:dst_ip}((%{DATA:dst_fwuser}))? (type %{INT:icmp_type}, code %{INT:icmp_code})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.329000-0400", :message=>"Adding pattern", "CISCOFW106015"=>"%{CISCO_ACTION:action} %{WORD:protocol} (%{DATA:policy_id}) from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{DATA:tcp_flags} on interface %{GREEDYDATA:interface}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.329000-0400", :message=>"Adding pattern", "CISCOFW106021"=>"%{CISCO_ACTION:action} %{WORD:protocol} reverse path check from %{IP:src_ip} to %{IP:dst_ip} on interface %{GREEDYDATA:interface}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.330000-0400", :message=>"Adding pattern", "CISCOFW106023"=>"%{CISCO_ACTION:action} %{WORD:protocol} src %{DATA:src_interface}:%{IP:src_ip}(/%{INT:src_port})?((%{DATA:src_fwuser}))? dst %{DATA:dst_interface}:%{IP:dst_ip}(/%{INT:dst_port})?((%{DATA:dst_fwuser}))?( (type %{INT:icmp_type}, code %{INT:icmp_code}))? by access-group %{NOTSPACE:policy_id} [%{DATA:hashcode1}, %{DATA:hashcode2}]", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.331000-0400", :message=>"Adding pattern", "CISCOFW106100"=>"access-list %{NOTSPACE:policy_id} %{CISCO_ACTION:action} %{WORD:protocol} %{DATA:src_interface}/%{IP:src_ip}(%{INT:src_port})((%{DATA:src_fwuser}))? -> %{DATA:dst_interface}/%{IP:dst_ip}(%{INT:dst_port})((%{DATA:src_fwuser}))? hit-cnt %{INT:hit_count} %{CISCO_INTERVAL:interval} [%{DATA:hashcode1}, %{DATA:hashcode2}]", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.331000-0400", :message=>"Adding pattern", "CISCOFW110002"=>"%{CISCO_REASON:reason} for %{WORD:protocol} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.332000-0400", :message=>"Adding pattern", "CISCOFW302010"=>"%{INT:connection_count} in use, %{INT:connection_count_max} most used", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.332000-0400", :message=>"Adding pattern", "CISCOFW302013_302014_302015_302016"=>"%{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port}( (%{IP:src_mapped_ip}/%{INT:src_mapped_port}))?((%{DATA:src_fwuser}))? to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}( (%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}))?((%{DATA:dst_fwuser}))?( duration %{TIME:duration} bytes %{INT:bytes})?(?: %{CISCO_REASON:reason})?( (%{DATA:user}))?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.333000-0400", :message=>"Adding pattern", "CISCOFW302020_302021"=>"%{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection for faddr %{IP:dst_ip}/%{INT:icmp_seq_num}(?:(%{DATA:fwuser}))? gaddr %{IP:src_xlated_ip}/%{INT:icmp_code_xlated} laddr %{IP:src_ip}/%{INT:icmp_code}( (%{DATA:user}))?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.333000-0400", :message=>"Adding pattern", "CISCOFW305011"=>"%{CISCO_ACTION:action} %{CISCO_XLATE_TYPE:xlate_type} %{WORD:protocol} translation from %{DATA:src_interface}:%{IP:src_ip}(/%{INT:src_port})?((%{DATA:src_fwuser}))? to %{DATA:src_xlated_interface}:%{IP:src_xlated_ip}/%{DATA:src_xlated_port}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.347000-0400", :message=>"Adding pattern", "CISCOFW313001_313004_313008"=>"%{CISCO_ACTION:action} %{WORD:protocol} type=%{INT:icmp_type}, code=%{INT:icmp_code} from %{IP:src_ip} on interface %{DATA:interface}( to %{IP:dst_ip})?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.351000-0400", :message=>"Adding pattern", "CISCOFW313005"=>"%{CISCO_REASON:reason} for %{WORD:protocol} error message: %{WORD:err_protocol} src %{DATA:err_src_interface}:%{IP:err_src_ip}((%{DATA:err_src_fwuser}))? dst %{DATA:err_dst_interface}:%{IP:err_dst_ip}((%{DATA:err_dst_fwuser}))? (type %{INT:err_icmp_type}, code %{INT:err_icmp_code}) on %{DATA:interface} interface. Original IP payload: %{WORD:protocol} src %{IP:orig_src_ip}/%{INT:orig_src_port}((%{DATA:orig_src_fwuser}))? dst %{IP:orig_dst_ip}/%{INT:orig_dst_port}((%{DATA:orig_dst_fwuser}))?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.352000-0400", :message=>"Adding pattern", "CISCOFW321001"=>"Resource '%{WORD:resource_name}' limit of %{POSINT:resource_limit} reached for system", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.364000-0400", :message=>"Adding pattern", "CISCOFW402117"=>"%{WORD:protocol}: Received a non-IPSec packet (protocol= %{WORD:orig_protocol}) from %{IP:src_ip} to %{IP:dst_ip}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.365000-0400", :message=>"Adding pattern", "CISCOFW402119"=>"%{WORD:protocol}: Received an %{WORD:orig_protocol} packet (SPI= %{DATA:spi}, sequence number= %{DATA:seq_num}) from %{IP:src_ip} (user= %{DATA:user}) to %{IP:dst_ip} that failed anti-replay checking", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.365000-0400", :message=>"Adding pattern", "CISCOFW419001"=>"%{CISCO_ACTION:action} %{WORD:protocol} packet from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}, reason: %{GREEDYDATA:reason}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.366000-0400", :message=>"Adding pattern", "CISCOFW419002"=>"%{CISCO_REASON:reason} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port} with different initial sequence number", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.366000-0400", :message=>"Adding pattern", "CISCOFW500004"=>"%{CISCO_REASON:reason} for protocol=%{WORD:protocol}, from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.367000-0400", :message=>"Adding pattern", "CISCOFW602303_602304"=>"%{WORD:protocol}: An %{CISCO_DIRECTION:direction} %{GREEDYDATA:tunnel_type} SA (SPI= %{DATA:spi}) between %{IP:src_ip} and %{IP:dst_ip} (user= %{DATA:user}) has been %{CISCO_ACTION:action}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.368000-0400", :message=>"Adding pattern", "CISCOFW710001_710002_710003_710005_710006"=>"%{WORD:protocol} (?:request|access) %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.368000-0400", :message=>"Adding pattern", "CISCOFW713172"=>"Group = %{GREEDYDATA:group}, IP = %{IP:srcip}, Automatic NAT Detection Status:\s+Remote end\s%{DATA:is_remote_natted}\sbehind a NAT device\s+This\s+end\s%{DATA:is_local_natted}\s_behind a NAT device", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.369000-0400", :message=>"Adding pattern", "CISCOFW733100"=>"[\s%{DATA:droptype}\s] drop %{DATA:drop_rate_id} exceeded. Current burst rate is %{INT:drop_rate_current_burst} per second, max configured rate is %{INT:drop_rate_max_burst}; Current average rate is %{INT:drop_rate_current_avg} per second, max configured rate is %{INT:drop_rate_max_avg}; Cumulative total count is %{INT:drop_total_count}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.369000-0400", :message=>"Adding pattern", "SHOREWALL"=>"(%{SYSLOGTIMESTAMP:timestamp}) (%{WORD:nf_host}) kernel:._Shorewall:(%{WORD:nf_action1})?:(%{WORD:nf_action2})?._IN=(%{USERNAME:nf_ininterface})?.(OUT= *MAC=(%{COMMONMAC:nf_dst_mac}):(%{COMMONMAC:nf_src_mac})?|OUT=%{USERNAME:nf_out_interface})._SRC=(%{IPV4:nf_src_ip})._DST=(%{IPV4:nf_dst_ip})._LEN=(%{WORD:nf_len}).?_TOS=(%{WORD:nf_tos}).?_PREC=(%{WORD:nf_prec}).?_TTL=(%{INT:nf_ttl}).?_ID=(%{INT:nf_id}).?_PROTO=(%{WORD:nf_protocol}).?_SPT=(%{INT:nf_src_port}?._DPT=%{INT:nf_dstport}?.)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.379000-0400", :message=>"Adding pattern", "USERNAME"=>"[a-zA-Z0-9.-]+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.379000-0400", :message=>"Adding pattern", "USER"=>"%{USERNAME}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.380000-0400", :message=>"Adding pattern", "INT"=>"(?:[+-]?(?:[0-9]+))", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.380000-0400", :message=>"Adding pattern", "BASE10NUM"=>"(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:.[0-9]+)?)|(?:.[0-9]+)))", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.381000-0400", :message=>"Adding pattern", "NUMBER"=>"(?:%{BASE10NUM})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.381000-0400", :message=>"Adding pattern", "BASE16NUM"=>"(?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.402000-0400", :message=>"Adding pattern", "BASE16FLOAT"=>"\b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:.[0-9A-Fa-f])?)|(?:.[0-9A-Fa-f]+)))\b", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.403000-0400", :message=>"Adding pattern", "POSINT"=>"\b(?:[1-9][0-9])\b", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.404000-0400", :message=>"Adding pattern", "NONNEGINT"=>"\b(?:[0-9]+)\b", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.404000-0400", :message=>"Adding pattern", "WORD"=>"\b\w+\b", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.405000-0400", :message=>"Adding pattern", "NOTSPACE"=>"\S+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.405000-0400", :message=>"Adding pattern", "SPACE"=>"\s", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.406000-0400", :message=>"Adding pattern", "DATA"=>".?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.406000-0400", :message=>"Adding pattern", "GREEDYDATA"=>".", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.407000-0400", :message=>"Adding pattern", "QUOTEDSTRING"=>"(?>(?<!\)(?>\"(?>\.|[^\\"]+)+\"|\"\"|(?>'(?>\.|[^\']+)+')|''|(?>(?>\\\\.|[^\\\\]+)+`)|``))", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.408000-0400", :message=>"Adding pattern", "UUID"=>"[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.408000-0400", :message=>"Adding pattern", "MAC"=>"(?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.409000-0400", :message=>"Adding pattern", "CISCOMAC"=>"(?:(?:[A-Fa-f0-9]{4}.){2}[A-Fa-f0-9]{4})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.409000-0400", :message=>"Adding pattern", "WINDOWSMAC"=>"(?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.410000-0400", :message=>"Adding pattern", "COMMONMAC"=>"(?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.410000-0400", :message=>"Adding pattern", "IPV6"=>"((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.411000-0400", :message=>"Adding pattern", "IPV4"=>"(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})...)(?![0-9])", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.411000-0400", :message=>"Adding pattern", "IP"=>"(?:%{IPV6}|%{IPV4})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.412000-0400", :message=>"Adding pattern", "HOSTNAME"=>"\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.413000-0400", :message=>"Adding pattern", "HOST"=>"%{HOSTNAME}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.413000-0400", :message=>"Adding pattern", "IPORHOST"=>"(?:%{HOSTNAME}|%{IP})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.419000-0400", :message=>"Adding pattern", "HOSTPORT"=>"%{IPORHOST}:%{POSINT}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.420000-0400", :message=>"Adding pattern", "PATH"=>"(?:%{UNIXPATH}|%{WINPATH})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.420000-0400", :message=>"Adding pattern", "UNIXPATH"=>"(?>/(?>[\w%!$@:.,~-]+|\.)_)+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.421000-0400", :message=>"Adding pattern", "TTY"=>"(?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.421000-0400", :message=>"Adding pattern", "WINPATH"=>"(?>[A-Za-z]+:|\)(?:\[^\?]_)+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.422000-0400", :message=>"Adding pattern", "URIPROTO"=>"[A-Za-z]+(+[A-Za-z+]+)?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.422000-0400", :message=>"Adding pattern", "URIHOST"=>"%{IPORHOST}(?::%{POSINT:port})?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.423000-0400", :message=>"Adding pattern", "URIPATH"=>"(?:/[A-Za-z0-9$.+!'(){},~:;=@#%-])+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.423000-0400", :message=>"Adding pattern", "URIPARAM"=>"\?[A-Za-z0-9$.+!'|(){},~@#%&/=:;?-[]]", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.424000-0400", :message=>"Adding pattern", "URIPATHPARAM"=>"%{URIPATH}(?:%{URIPARAM})?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.424000-0400", :message=>"Adding pattern", "URI"=>"%{URIPROTO}://(?:%{USER}(?::[^@])?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.425000-0400", :message=>"Adding pattern", "MONTH"=>"\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.426000-0400", :message=>"Adding pattern", "MONTHNUM"=>"(?:0?[1-9]|1[0-2])", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.426000-0400", :message=>"Adding pattern", "MONTHNUM2"=>"(?:0[1-9]|1[0-2])", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.427000-0400", :message=>"Adding pattern", "MONTHDAY"=>"(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.427000-0400", :message=>"Adding pattern", "DAY"=>"(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.428000-0400", :message=>"Adding pattern", "YEAR"=>"(?>\d\d){1,2}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.428000-0400", :message=>"Adding pattern", "HOUR"=>"(?:2[0123]|[01]?[0-9])", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.429000-0400", :message=>"Adding pattern", "MINUTE"=>"(?:[0-5][0-9])", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.463000-0400", :message=>"Adding pattern", "SECOND"=>"(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.464000-0400", :message=>"Adding pattern", "TIME"=>"(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.465000-0400", :message=>"Adding pattern", "DATE_US"=>"%{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.465000-0400", :message=>"Adding pattern", "DATE_EU"=>"%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.466000-0400", :message=>"Adding pattern", "ISO8601_TIMEZONE"=>"(?:Z|[+-]%{HOUR}(?::?%{MINUTE}))", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.467000-0400", :message=>"Adding pattern", "ISO8601_SECOND"=>"(?:%{SECOND}|60)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.467000-0400", :message=>"Adding pattern", "TIMESTAMP_ISO8601"=>"%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.468000-0400", :message=>"Adding pattern", "DATE"=>"%{DATE_US}|%{DATE_EU}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.468000-0400", :message=>"Adding pattern", "DATESTAMP"=>"%{DATE}[- ]%{TIME}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.469000-0400", :message=>"Adding pattern", "TZ"=>"(?:[PMCE][SD]T|UTC)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.469000-0400", :message=>"Adding pattern", "DATESTAMP_RFC822"=>"%{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.470000-0400", :message=>"Adding pattern", "DATESTAMP_RFC2822"=>"%{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.470000-0400", :message=>"Adding pattern", "DATESTAMP_OTHER"=>"%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.471000-0400", :message=>"Adding pattern", "DATESTAMP_EVENTLOG"=>"%{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.471000-0400", :message=>"Adding pattern", "SYSLOGTIMESTAMP"=>"%{MONTH} +%{MONTHDAY} %{TIME}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.472000-0400", :message=>"Adding pattern", "PROG"=>"[\x21-\x5a\x5c\x5e-\x7e]+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.472000-0400", :message=>"Adding pattern", "SYSLOGPROG"=>"%{PROG:program}(?:[%{POSINT:pid}])?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.473000-0400", :message=>"Adding pattern", "SYSLOGHOST"=>"%{IPORHOST}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.473000-0400", :message=>"Adding pattern", "SYSLOGFACILITY"=>"<%{NONNEGINT:facility}.%{NONNEGINT:priority}>", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.474000-0400", :message=>"Adding pattern", "HTTPDATE"=>"%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.478000-0400", :message=>"Adding pattern", "QS"=>"%{QUOTEDSTRING}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.479000-0400", :message=>"Adding pattern", "SYSLOGBASE"=>"%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.479000-0400", :message=>"Adding pattern", "COMMONAPACHELOG"=>"%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.480000-0400", :message=>"Adding pattern", "COMBINEDAPACHELOG"=>"%{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.480000-0400", :message=>"Adding pattern", "LOGLEVEL"=>"([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.502000-0400", :message=>"Adding pattern", "HAPROXYTIME"=>"(?!<[0-9])%{HOUR:haproxy_hour}:%{MINUTE:haproxy_minute}(?::%{SECOND:haproxy_second})(?![0-9])", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.502000-0400", :message=>"Adding pattern", "HAPROXYDATE"=>"%{MONTHDAY:haproxy_monthday}/%{MONTH:haproxy_month}/%{YEAR:haproxy_year}:%{HAPROXYTIME:haproxy_time}.%{INT:haproxy_milliseconds}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.503000-0400", :message=>"Adding pattern", "HAPROXYCAPTUREDREQUESTHEADERS"=>"%{DATA:captured_request_headers}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.505000-0400", :message=>"Adding pattern", "HAPROXYCAPTUREDRESPONSEHEADERS"=>"%{DATA:captured_response_headers}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.506000-0400", :message=>"Adding pattern", "HAPROXYHTTPBASE"=>"%{IP:client_ip}:%{INT:client_port} [%{HAPROXYDATE:accept_date}] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} ({%{HAPROXYCAPTUREDREQUESTHEADERS}})?( )?({%{HAPROXYCAPTUREDRESPONSEHEADERS}})?( )?\"(|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:httpuser}(?::[^@])?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?\"", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.506000-0400", :message=>"Adding pattern", "HAPROXYHTTP"=>"%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{HAPROXYHTTPBASE}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.507000-0400", :message=>"Adding pattern", "HAPROXYTCP"=>"%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} [%{HAPROXYDATE:accept_date}] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_queue}/%{INT:time_backend_connect}/%{NOTSPACE:time_duration} %{NOTSPACE:bytes_read} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.512000-0400", :message=>"Adding pattern", "JAVACLASS"=>"(?:[a-zA-Z$][a-zA-Z$0-9].)[a-zA-Z$][a-zA-Z$0-9]", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.513000-0400", :message=>"Adding pattern", "JAVAFILE"=>"(?:[A-Za-z0-9. -]+)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.513000-0400", :message=>"Adding pattern", "JAVAMETHOD"=>"(?:()|[a-zA-Z$][a-zA-Z$_0-9]*)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.514000-0400", :message=>"Adding pattern", "JAVASTACKTRACEPART"=>"%{SPACE}at %{JAVACLASS:class}.%{JAVAMETHOD:method}(%{JAVAFILE:file}(?::%{NUMBER:line})?)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.514000-0400", :message=>"Adding pattern", "JAVATHREAD"=>"(?:[A-Z]{2}-Processor[\d]+)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.515000-0400", :message=>"Adding pattern", "JAVACLASS"=>"(?:[a-zA-Z0-9-]+.)+[A-Za-z0-9$]+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.515000-0400", :message=>"Adding pattern", "JAVAFILE"=>"(?:[A-Za-z0-9.-]+)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.519000-0400", :message=>"Adding pattern", "JAVASTACKTRACEPART"=>"at %{JAVACLASS:class}.%{WORD:method}(%{JAVAFILE:file}:%{NUMBER:line})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.520000-0400", :message=>"Adding pattern", "JAVALOGMESSAGE"=>"(.)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.521000-0400", :message=>"Adding pattern", "CATALINA_DATESTAMP"=>"%{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.521000-0400", :message=>"Adding pattern", "TOMCAT_DATESTAMP"=>"20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.522000-0400", :message=>"Adding pattern", "CATALINALOG"=>"%{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.522000-0400", :message=>"Adding pattern", "TOMCATLOG"=>"%{TOMCAT_DATESTAMP:timestamp} | %{LOGLEVEL:level} | %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.526000-0400", :message=>"Adding pattern", "RT_FLOW_EVENT"=>"(RT_FLOW_SESSION_CREATE|RT_FLOW_SESSION_CLOSE|RT_FLOW_SESSION_DENY)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.526000-0400", :message=>"Adding pattern", "RT_FLOW1"=>"%{RT_FLOWEVENT:event}: %{GREEDYDATA:close-reason}: %{IP:src-ip}/%{DATA:src-port}->%{IP:dst-ip}/%{DATA:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{DATA:nat-src-port}->%{IP:nat-dst-ip}/%{DATA:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} \d+(%{DATA:sent}) \d+(%{DATA:received}) %{INT:elapsed-time} .", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.527000-0400", :message=>"Adding pattern", "RT_FLOW2"=>"%{RT_FLOWEVENT:event}: session created %{IP:src-ip}/%{DATA:src-port}->%{IP:dst-ip}/%{DATA:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{DATA:nat-src-port}->%{IP:nat-dst-ip}/%{DATA:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} .", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.527000-0400", :message=>"Adding pattern", "RT_FLOW3"=>"%{RT_FLOWEVENT:event}: session denied %{IP:src-ip}/%{DATA:src-port}->%{IP:dst-ip}/%{DATA:dst-port} %{DATA:service} %{INT:protocol-id}(\d) %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} .", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.550000-0400", :message=>"Adding pattern", "SYSLOG5424PRINTASCII"=>"[!-~]+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.551000-0400", :message=>"Adding pattern", "SYSLOGBASE2"=>"(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.551000-0400", :message=>"Adding pattern", "SYSLOGPAMSESSION"=>"%{SYSLOGBASE} (?=%{GREEDYDATA:message})%{WORD:pam_module}(%{DATA:pam_caller}): session %{WORD:pam_session_state} for user %{USERNAME:username}(?: by %{GREEDYDATA:pam_by})?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.552000-0400", :message=>"Adding pattern", "CRON_ACTION"=>"[A-Z ]+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.552000-0400", :message=>"Adding pattern", "CRONLOG"=>"%{SYSLOGBASE} (%{USER:user}) %{CRON_ACTION:action} (%{DATA:message})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.553000-0400", :message=>"Adding pattern", "SYSLOGLINE"=>"%{SYSLOGBASE2} %{GREEDYDATA:message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.553000-0400", :message=>"Adding pattern", "SYSLOG5424PRI"=>"<%{NONNEGINT:syslog5424_pri}>", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.554000-0400", :message=>"Adding pattern", "SYSLOG5424SD"=>"[%{DATA}]+", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.554000-0400", :message=>"Adding pattern", "SYSLOG5424BASE"=>"%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(-|%{SYSLOG5424PRINTASCII:syslog5424_app}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_proc}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_msgid}) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.555000-0400", :message=>"Adding pattern", "SYSLOG5424LINE"=>"%{SYSLOG5424BASE} +%{GREEDYDATA:syslog5424_msg}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.562000-0400", :message=>"Adding pattern", "MCOLLECTIVEAUDIT"=>"%{TIMESTAMP_ISO8601:timestamp}:", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.580000-0400", :message=>"Adding pattern", "MCOLLECTIVE"=>"., [%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}]%{SPACE}%{LOGLEVEL:event_level}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.581000-0400", :message=>"Adding pattern", "MCOLLECTIVEAUDIT"=>"%{TIMESTAMP_ISO8601:timestamp}:", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.582000-0400", :message=>"Adding pattern", "MONGO_LOG"=>"%{SYSLOGTIMESTAMP:timestamp} [%{WORD:component}] %{GREEDYDATA:message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.583000-0400", :message=>"Adding pattern", "MONGOQUERY"=>"{ (?<={ ).(?= } ntoreturn:) }", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.585000-0400", :message=>"Adding pattern", "MONGO_SLOWQUERY"=>"%{WORD} %{MONGO_WORDDASH:database}.%{MONGO_WORDDASH:collection} %{WORD}: %{MONGO_QUERY:query} %{WORD}:%{NONNEGINT:ntoreturn} %{WORD}:%{NONNEGINT:ntoskip} %{WORD}:%{NONNEGINT:nscanned}._nreturned:%{NONNEGINT:nreturned}..+ (?[0-9]+)ms", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.585000-0400", :message=>"Adding pattern", "MONGO_WORDDASH"=>"\b[\w-]+\b", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.585000-0400", :message=>"Adding pattern", "MONGO3_SEVERITY"=>"\w", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.586000-0400", :message=>"Adding pattern", "MONGO3_COMPONENT"=>"%{WORD}|-", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.586000-0400", :message=>"Adding pattern", "MONGO3_LOG"=>"%{TIMESTAMP_ISO8601:timestamp} %{MONGO3_SEVERITY:severity} %{MONGO3_COMPONENT:component}%{SPACE}(?:[%{DATA:context}])? %{GREEDYDATA:message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.614000-0400", :message=>"Adding pattern", "NAGIOSTIME"=>"[%{NUMBER:nagios_epoch}]", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.615000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_CURRENT_SERVICE_STATE"=>"CURRENT SERVICE STATE", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.615000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_CURRENT_HOST_STATE"=>"CURRENT HOST STATE", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.616000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_SERVICE_NOTIFICATION"=>"SERVICE NOTIFICATION", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.616000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_HOST_NOTIFICATION"=>"HOST NOTIFICATION", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.617000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_SERVICE_ALERT"=>"SERVICE ALERT", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.617000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_HOST_ALERT"=>"HOST ALERT", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.618000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_SERVICE_FLAPPING_ALERT"=>"SERVICE FLAPPING ALERT", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.618000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_HOST_FLAPPING_ALERT"=>"HOST FLAPPING ALERT", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.622000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT"=>"SERVICE DOWNTIME ALERT", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.623000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_HOST_DOWNTIME_ALERT"=>"HOST DOWNTIME ALERT", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.624000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_PASSIVE_SERVICE_CHECK"=>"PASSIVE SERVICE CHECK", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.624000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_PASSIVE_HOST_CHECK"=>"PASSIVE HOST CHECK", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.624000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_SERVICE_EVENT_HANDLER"=>"SERVICE EVENT HANDLER", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.625000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_HOST_EVENT_HANDLER"=>"HOST EVENT HANDLER", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.625000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_EXTERNAL_COMMAND"=>"EXTERNAL COMMAND", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.626000-0400", :message=>"Adding pattern", "NAGIOS_TYPE_TIMEPERIOD_TRANSITION"=>"TIMEPERIOD TRANSITION", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.626000-0400", :message=>"Adding pattern", "NAGIOS_EC_DISABLE_SVC_CHECK"=>"DISABLE_SVC_CHECK", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.627000-0400", :message=>"Adding pattern", "NAGIOS_EC_ENABLE_SVC_CHECK"=>"ENABLE_SVC_CHECK", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.627000-0400", :message=>"Adding pattern", "NAGIOS_EC_DISABLE_HOST_CHECK"=>"DISABLE_HOST_CHECK", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.628000-0400", :message=>"Adding pattern", "NAGIOS_EC_ENABLE_HOST_CHECK"=>"ENABLE_HOST_CHECK", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.628000-0400", :message=>"Adding pattern", "NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT"=>"PROCESS_SERVICE_CHECK_RESULT", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.629000-0400", :message=>"Adding pattern", "NAGIOS_EC_PROCESS_HOST_CHECK_RESULT"=>"PROCESS_HOST_CHECK_RESULT", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.629000-0400", :message=>"Adding pattern", "NAGIOS_EC_SCHEDULE_SERVICE_DOWNTIME"=>"SCHEDULE_SERVICE_DOWNTIME", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.630000-0400", :message=>"Adding pattern", "NAGIOS_EC_SCHEDULE_HOST_DOWNTIME"=>"SCHEDULE_HOST_DOWNTIME", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.630000-0400", :message=>"Adding pattern", "NAGIOS_WARNING"=>"Warning:%{SPACE}%{GREEDYDATA:nagios_message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.631000-0400", :message=>"Adding pattern", "NAGIOS_CURRENT_SERVICE_STATE"=>"%{NAGIOS_TYPE_CURRENT_SERVICE_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.631000-0400", :message=>"Adding pattern", "NAGIOS_CURRENT_HOST_STATE"=>"%{NAGIOS_TYPE_CURRENT_HOST_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.635000-0400", :message=>"Adding pattern", "NAGIOS_SERVICE_NOTIFICATION"=>"%{NAGIOS_TYPE_SERVICE_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.635000-0400", :message=>"Adding pattern", "NAGIOS_HOST_NOTIFICATION"=>"%{NAGIOS_TYPE_HOST_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.636000-0400", :message=>"Adding pattern", "NAGIOS_SERVICE_ALERT"=>"%{NAGIOS_TYPE_SERVICE_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.636000-0400", :message=>"Adding pattern", "NAGIOS_HOST_ALERT"=>"%{NAGIOS_TYPE_HOST_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.637000-0400", :message=>"Adding pattern", "NAGIOS_SERVICE_FLAPPING_ALERT"=>"%{NAGIOS_TYPE_SERVICE_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.637000-0400", :message=>"Adding pattern", "NAGIOS_HOST_FLAPPING_ALERT"=>"%{NAGIOS_TYPE_HOST_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.638000-0400", :message=>"Adding pattern", "NAGIOS_SERVICE_DOWNTIME_ALERT"=>"%{NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.638000-0400", :message=>"Adding pattern", "NAGIOS_HOST_DOWNTIME_ALERT"=>"%{NAGIOS_TYPE_HOST_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.639000-0400", :message=>"Adding pattern", "NAGIOS_PASSIVE_SERVICE_CHECK"=>"%{NAGIOS_TYPE_PASSIVE_SERVICE_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.639000-0400", :message=>"Adding pattern", "NAGIOS_PASSIVE_HOST_CHECK"=>"%{NAGIOS_TYPE_PASSIVE_HOST_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.640000-0400", :message=>"Adding pattern", "NAGIOS_SERVICE_EVENT_HANDLER"=>"%{NAGIOS_TYPE_SERVICE_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.640000-0400", :message=>"Adding pattern", "NAGIOS_HOST_EVENT_HANDLER"=>"%{NAGIOS_TYPE_HOST_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.641000-0400", :message=>"Adding pattern", "NAGIOS_TIMEPERIOD_TRANSITION"=>"%{NAGIOS_TYPE_TIMEPERIOD_TRANSITION:nagios_type}: %{DATA:nagios_service};%{DATA:nagios_unknown1};%{DATA:nagios_unknown2}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.641000-0400", :message=>"Adding pattern", "NAGIOS_EC_LINE_DISABLE_SVC_CHECK"=>"%{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.642000-0400", :message=>"Adding pattern", "NAGIOS_EC_LINE_DISABLE_HOST_CHECK"=>"%{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.642000-0400", :message=>"Adding pattern", "NAGIOS_EC_LINE_ENABLE_SVC_CHECK"=>"%{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.643000-0400", :message=>"Adding pattern", "NAGIOS_EC_LINE_ENABLE_HOST_CHECK"=>"%{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.643000-0400", :message=>"Adding pattern", "NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT"=>"%{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.644000-0400", :message=>"Adding pattern", "NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT"=>"%{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_HOST_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.644000-0400", :message=>"Adding pattern", "NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME"=>"%{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_SCHEDULE_HOST_DOWNTIME:nagios_command};%{DATA:nagios_hostname};%{NUMBER:nagios_start_time};%{NUMBER:nagios_end_time};%{NUMBER:nagios_fixed};%{NUMBER:nagios_trigger_id};%{NUMBER:nagios_duration};%{DATA:author};%{DATA:comment}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.663000-0400", :message=>"Adding pattern", "NAGIOSLOGLINE"=>"%{NAGIOSTIME} (?:%{NAGIOS_WARNING}|%{NAGIOS_CURRENT_SERVICE_STATE}|%{NAGIOS_CURRENT_HOST_STATE}|%{NAGIOS_SERVICE_NOTIFICATION}|%{NAGIOS_HOST_NOTIFICATION}|%{NAGIOS_SERVICE_ALERT}|%{NAGIOS_HOST_ALERT}|%{NAGIOS_SERVICE_FLAPPING_ALERT}|%{NAGIOS_HOST_FLAPPING_ALERT}|%{NAGIOS_SERVICE_DOWNTIME_ALERT}|%{NAGIOS_HOST_DOWNTIME_ALERT}|%{NAGIOS_PASSIVE_SERVICE_CHECK}|%{NAGIOS_PASSIVE_HOST_CHECK}|%{NAGIOS_SERVICE_EVENT_HANDLER}|%{NAGIOS_HOST_EVENT_HANDLER}|%{NAGIOS_TIMEPERIOD_TRANSITION}|%{NAGIOS_EC_LINE_DISABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_ENABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_DISABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_ENABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT}|%{NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT}|%{NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.679000-0400", :message=>"Adding pattern", "POSTGRESQL"=>"%{DATESTAMP:timestamp} %{TZ} %{DATA:user_id} %{GREEDYDATA:connection_id} %{POSINT:pid}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.682000-0400", :message=>"Adding pattern", "RUUID"=>"\h{32}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.682000-0400", :message=>"Adding pattern", "RCONTROLLER"=>"(?[^#]+)#(?\w+)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.683000-0400", :message=>"Adding pattern", "RAILS3HEAD"=>"(?m)Started %{WORD:verb} \"%{URIPATHPARAM:request}\" for %{IPORHOST:clientip} at (?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE})", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.683000-0400", :message=>"Adding pattern", "RPROCESSING"=>"\W_Processing by %{RCONTROLLER} as (?\S+)(?:\WParameters: {%{DATA:params}}\W)?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.684000-0400", :message=>"Adding pattern", "RAILS3FOOT"=>"Completed %{NUMBER:response}%{DATA} in %{NUMBER:totalms}ms %{RAILS3PROFILE}%{GREEDYDATA}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.684000-0400", :message=>"Adding pattern", "RAILS3PROFILE"=>"(?:(Views: %{NUMBER:viewms}ms | ActiveRecord: %{NUMBER:activerecordms}ms|(ActiveRecord: %{NUMBER:activerecordms}ms)?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"addpattern"} {:timestamp=>"2015-10-27T08:27:51.685000-0400", :message=>"Adding pattern", "RAILS3"=>"%{RAILS3HEAD}(?:%{RPROCESSING})?(?(?:%{DATA}\n))(?:%{RAILS3FOOT})?", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.722000-0400", :message=>"Adding pattern", "REDISTIMESTAMP"=>"%{MONTHDAY} %{MONTH} %{TIME}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.723000-0400", :message=>"Adding pattern", "REDISLOG"=>"[%{POSINT:pid}] %{REDISTIMESTAMP:timestamp} \* ", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.724000-0400", :message=>"Adding pattern", "RUBY_LOGLEVEL"=>"(?:DEBUG|FATAL|ERROR|WARN|INFO)", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.725000-0400", :message=>"Adding pattern", "RUBY_LOGGER"=>"[DFEWI], [%{TIMESTAMPISO8601:timestamp} #%{POSINT:pid}] %{RUBY_LOGLEVEL:loglevel} -- +%{DATA:progname}: %{GREEDYDATA:message}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"} {:timestamp=>"2015-10-27T08:27:51.726000-0400", :message=>"replacementpattern => (?POSINT:priority\b(?:[1-9][0-9])\b)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.727000-0400", :message=>"replacement_pattern => (?:%{SYSLOGBASE2} %{GREEDYDATA:message})", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.728000-0400", :message=>"replacement_pattern => (?:(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.729000-0400", :message=>"replacement_pattern => (?SYSLOGTIMESTAMP:timestamp%{MONTH} +%{MONTHDAY} %{TIME})", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.730000-0400", :message=>"replacement_pattern => (?:\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.730000-0400", :message=>"replacement_pattern => (?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.734000-0400", :message=>"replacement_pattern => (?:(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.734000-0400", :message=>"replacement_pattern => (?:(?:2[0123]|[01]?[0-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.735000-0400", :message=>"replacement_pattern => (?:(?:[0-5][0-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.735000-0400", :message=>"replacement_pattern => (?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.736000-0400", :message=>"replacement_pattern => (?%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.736000-0400", :message=>"replacement_pattern => (?:(?>\d\d){1,2})", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.737000-0400", :message=>"replacement_pattern => (?:(?:0?[1-9]|1[0-2]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.737000-0400", :message=>"replacement_pattern => (?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.738000-0400", :message=>"replacement_pattern => (?:(?:2[0123]|[01]?[0-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.738000-0400", :message=>"replacement_pattern => (?:(?:[0-5][0-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.739000-0400", :message=>"replacement_pattern => (?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.739000-0400", :message=>"replacement_pattern => (?:(?:Z|[+-]%{HOUR}(?::?%{MINUTE})))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.740000-0400", :message=>"replacement_pattern => (?:(?:2[0123]|[01]?[0-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.740000-0400", :message=>"replacement_pattern => (?:(?:[0-5][0-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.741000-0400", :message=>"replacement_pattern => (?:<%{NONNEGINT:facility}.%{NONNEGINT:priority}>)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.741000-0400", :message=>"replacement_pattern => (?NONNEGINT:facility\b(?:[0-9]+)\b)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.742000-0400", :message=>"replacement_pattern => (?NONNEGINT:priority\b(?:[0-9]+)\b)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.742000-0400", :message=>"replacement_pattern => (?SYSLOGHOST:logsource%{IPORHOST})", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.743000-0400", :message=>"replacement_pattern => (?:(?:%{HOSTNAME}|%{IP}))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.743000-0400", :message=>"replacementpattern => (?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.750000-0400", :message=>"replacement_pattern => (?:(?:%{IPV6}|%{IPV4}))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.751000-0400", :message=>"replacement_pattern => (?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.752000-0400", :message=>"replacement_pattern => (?:(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})...)(?![0-9]))", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.752000-0400", :message=>"replacement_pattern => (?:%{PROG:program}(?:[%{POSINT:pid}])?)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.753000-0400", :message=>"replacement_pattern => (?PROG:program[\x21-\x5a\x5c\x5e-\x7e]+)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.753000-0400", :message=>"replacementpattern => (?POSINT:pid\b(?:[1-9][0-9])\b)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.754000-0400", :message=>"replacementpattern => (?GREEDYDATA:message.)", :level=>:debug, :file=>"grok-pure.rb", :line=>"121", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.756000-0400", :message=>"Grok compiled OK", :pattern=>"<%{POSINT:priority}>%{SYSLOGLINE}", :expandedpattern=>"<(?POSINT:priority\b(?:[1-9][0-9])\b)>(?:(?:(?:(?SYSLOGTIMESTAMP:timestamp(?:\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b) +(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?:(?!<[0-9])(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))(?![0-9])))|(?(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))T :?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|+-(?::?(?:(?:[0-5][0-9])))))?)) (?:(?:<(?NONNEGINT:facility\b(?:[0-9]+)\b).(?NONNEGINT:priority\b(?:[0-9]+)\b)>) )?(?SYSLOGHOST:logsource(?:(?:(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b))|(?:(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})...)(?![0-9]))))))) (?:(?PROG:program[\x21-\x5a\x5c\x5e-\x7e]+)(?:[(?POSINT:pid\b(?:[1-9][0-9])\b)])?):) (?GREEDYDATA:message.*))", :level=>:debug, :file=>"grok-pure.rb", :line=>"128", :method=>"compile"} {:timestamp=>"2015-10-27T08:27:51.818000-0400", :message=>"Adding type with date config", :type=>"", :field=>"timestamp", :format=>"MMM d HH:mm:ss", :level=>:debug, :file=>"logstash/filters/date.rb", :line=>"193", :method=>"setupMatcher"} {:timestamp=>"2015-10-27T08:27:51.819000-0400", :message=>"Adding type with date config", :type=>"", :field=>"timestamp", :format=>"MMM dd HH:mm:ss", :level=>:debug, :file=>"logstash/filters/date.rb", :line=>"193", :method=>"setupMatcher"} {:timestamp=>"2015-10-27T08:27:51.821000-0400", :message=>"Adding type with date config", :type=>"", :field=>"timestamp", :format=>"ISO8601", :level=>:debug, :file=>"logstash/filters/date.rb", :line=>"193", :method=>"setupMatcher"} {:timestamp=>"2015-10-27T08:27:51.865000-0400", :message=>"Normalizing http path", :path=>"/", :normalized=>"/", :level=>:debug, :file=>"logstash/outputs/elasticsearch.rb", :line=>"353", :method=>"register"} {:timestamp=>"2015-10-27T08:27:51.871000-0400", :message=>"Starting syslog udp listener", :address=>"0.0.0.0:5514", :level=>:info, :file=>"logstash/inputs/syslog.rb", :line=>"131", :method=>"udp_listener"} {:timestamp=>"2015-10-27T08:27:51.890000-0400", :message=>"Starting syslog tcp listener", :address=>"0.0.0.0:5514", :level=>:info, :file=>"logstash/inputs/syslog.rb", :line=>"151", :method=>"tcp_listener"}

joshusre commented 8 years ago

Should I try to attach the file? It wouldn't let me but maybe i did it wrong?

RomanHotsiy commented 8 years ago

@joshusre I had the same issue. The problem was in access rights. Check out owner of files in your /var/log/mylogs dir. And add logstash user to the corresponding group. This solved the problem for me.

It is not an issue of logstash, so @jsvd you can close the issue.

joshusre commented 8 years ago

Sweet, I will monday.

If this isn't something that Logstash does on install, and isn't really documented anywhere "that I could find anyway", what do we call this and how do we prevent others from experiencing it?

On Sat, Oct 31, 2015 at 3:54 AM, Roman Hotsiy notifications@github.com wrote:

@joshusre https://github.com/joshusre I had the same issue. The problem was in access rights. Check out owner of files in your /var/log/mylogs dir. And add logstash user to the corresponding group. This solved the problem for me.

It is not an issue of logstash, so @jsvd https://github.com/jsvd you can close the issue.

— Reply to this email directly or view it on GitHub https://github.com/elastic/logstash/issues/4075#issuecomment-152716021.